Sep 17 15:39:11.327427 ip-10-0-133-29 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:39:11.703556 ip-10-0-133-29 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:39:11.703556 ip-10-0-133-29 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:39:11.703556 ip-10-0-133-29 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:39:11.703556 ip-10-0-133-29 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:39:11.703556 ip-10-0-133-29 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:39:11.704632 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.704544    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:39:11.709542 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709513    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:11.709542 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709541    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:11.709542 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709546    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:11.709542 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709549    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709554    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709557    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709560    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709563    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709566    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709569    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709572    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709574    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709577    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709579    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709582    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709587    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709597    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709600    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709603    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709606    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709609    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709612    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709615    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:11.709672 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709618    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709621    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709623    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709626    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709629    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709631    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709634    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709638    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709641    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709644    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709647    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709650    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709654    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709656    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709659    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709662    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709665    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709668    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709670    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709673    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:11.710196 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709676    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709678    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709681    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709684    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709687    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709690    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709692    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709695    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709697    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709700    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709702    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709705    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709707    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709710    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709712    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709715    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709718    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709720    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709723    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709726    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:11.710782 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709728    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709731    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709734    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709736    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709740    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709743    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709746    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709749    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709752    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709756    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709758    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709762    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709766    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709770    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709773    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709776    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709780    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709783    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709785    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:11.711281 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709788    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709791    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709793    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709796    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709798    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709801    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709803    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709806    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709808    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709811    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709814    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709817    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709819    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709822    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709826    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709828    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.709831    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710612    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710619    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:11.711738 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710623    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710626    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710630    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710633    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710635    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710638    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710643    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710646    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710649    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710652    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710655    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710660    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710663    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710665    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710668    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710671    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710674    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710676    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710679    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:11.712216 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710681    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710684    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710686    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710689    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710691    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710694    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710697    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710700    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710702    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710705    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710707    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710710    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710713    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710716    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710719    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710722    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710724    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710727    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710729    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710732    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:11.712684 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710734    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710737    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710739    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710742    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710745    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710747    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710750    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710752    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710755    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710758    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710760    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710763    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710765    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710767    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710769    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710773    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710776    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710779    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710782    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:11.713203 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710785    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710789    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710792    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710797    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710800    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710803    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710805    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710808    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710811    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710814    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710816    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710819    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710821    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710824    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710826    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710829    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710831    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710834    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710836    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710839    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:11.713669 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710841    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710844    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710847    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710849    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710852    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710854    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710857    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710859    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710862    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710864    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710867    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710870    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710872    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710875    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710877    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710880    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710883    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710885    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.710888    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711284    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:11.714184 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711289    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711292    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711295    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711298    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711300    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711303    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711306    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711308    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711311    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711314    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711317    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711319    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711322    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711324    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711327    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711330    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711332    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711335    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711338    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:11.714670 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711341    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711343    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711346    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711348    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711351    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711354    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711357    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711359    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711362    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711367    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711370    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711372    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711375    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711378    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711380    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711383    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711385    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711388    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711390    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711392    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:11.715136 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711395    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711398    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711400    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711403    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711405    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711408    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711412    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711415    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711419    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711421    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711423    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711426    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711428    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711432    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711434    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711437    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711440    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711443    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711445    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:11.715643 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711448    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711452    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711457    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711460    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711463    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711466    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711469    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711471    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711474    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711476    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711479    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711481    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711486    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711489    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711492    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711495    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711498    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711500    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711503    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:11.716122 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711506    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711509    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711511    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711514    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711516    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711519    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711521    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711524    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711527    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711529    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711532    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711535    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711538    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711540    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711543    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711550    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711552    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711555    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711558    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711560    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:11.716579 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.711563    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711644    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711680    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711686    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711691    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711695    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711698    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711702    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711706    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711710    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711713    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711717    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711720    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711724    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711727    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711730    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711733    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711736    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711739    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711743    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711746    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711753    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711756    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711760    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:39:11.717120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711763    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711766    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711769    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711773    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711776    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711779    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711782    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711785    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711788    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711797    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711800    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711802    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711806    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711808    2575 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711811    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711816    2575 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711819    2575 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711821    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711824    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711827    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711831    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711834    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711837    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711841    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711843    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:39:11.717692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711846    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711849    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711852    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711855    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711859    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711862    2575 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711865    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711869    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711872    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711875    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711878    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711881    2575 flags.go:64] FLAG: --help="false"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711884    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-133-29.ec2.internal"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711887    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711890    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711894    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711897    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711900    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711903    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711906    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711909    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711911    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711915    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711918    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:39:11.718326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711921    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711924    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711927    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711930    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711932    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711935    2575 flags.go:64] FLAG: --lock-file=""
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711939    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711942    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711944    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711950    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711952    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711955    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711959    2575 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711962    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711965    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711968    2575 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711971    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711975    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711979    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711983    2575 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711986    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711989    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711992    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711995    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.711998    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:39:11.718929 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712001    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712004    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712012    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712015    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712018    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712021    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712024    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712030    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712033    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712036    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712039    2575 flags.go:64] FLAG: --port="10250"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712042    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712045    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d9863e1b4885875"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712048    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712051    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712054    2575 flags.go:64] FLAG: --register-node="true"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712057    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712061    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712064    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712066    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712070    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712088    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712092    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712095    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712097    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:39:11.719547 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712100    2575 flags.go:64] FLAG: --runonce="false"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712103    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712106    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712109    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712112    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712114    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712117    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712121    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712124    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712126    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712129    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712132    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712135    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712138    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712141    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712144    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712150    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712153    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712155    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712160    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712163    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712165    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712169    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712172    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712175    2575 flags.go:64] FLAG: --v="2"
Sep 17 15:39:11.720198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712179    2575 flags.go:64] FLAG: --version="false"
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712183    2575 flags.go:64] FLAG: --vmodule=""
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712188    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.712191    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712306    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712311    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712314    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712317    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712320    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712323    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712326    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712328    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712331    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712333    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712336    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712339    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712341    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712344    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712346    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712349    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:11.720822 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712352    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712354    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712357    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712359    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712362    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712365    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712367    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712370    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712372    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712375    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712388    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712391    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712393    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712396    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712400    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712402    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712406    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712411    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712414    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:11.721309 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712417    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712420    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712423    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712426    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712428    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712431    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712434    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712436    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712439    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712441    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712444    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712446    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712449    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712451    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712454    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712456    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712459    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712462    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712464    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712467    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:11.721771 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712470    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712474    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712478    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712481    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712489    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712493    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712495    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712498    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712501    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712504    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712507    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712510    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712513    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712516    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712518    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712521    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712523    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712526    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712528    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:11.722344 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712531    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712534    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712536    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712539    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712543    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712546    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712549    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712552    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712555    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712558    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712561    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712563    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712566    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712568    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712571    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712574    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712576    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712579    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712587    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:11.722821 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712590    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712594    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712597    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712600    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712604    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.712607    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.713328    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.720733    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.720750    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.721502    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.721510    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721569    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721574    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721578    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721581    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:11.723300 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721584    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721587    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721590    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721593    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721596    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721598    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721601    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721603    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721606    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721609    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721611    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721614    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721617    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721619    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721622    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721624    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721627    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721630    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721632    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:11.723712 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721638    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721641    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721644    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721648    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721651    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721654    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721658    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721661    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721664    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721666    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721669    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721672    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721675    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721677    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721680    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721682    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721685    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721688    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721692    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:11.724192 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721696    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721699    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721701    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721704    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721707    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721711    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721713    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721716    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721719    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721721    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721724    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721727    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721730    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721733    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721735    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721738    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721740    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721743    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721746    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721749    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:11.724647 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721752    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721754    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721757    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721759    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721762    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721764    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721767    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721769    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721772    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721774    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721777    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721780    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721782    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721785    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721787    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721790    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721792    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721795    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721797    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721800    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:11.725149 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721802    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721805    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721808    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721810    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721813    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721816    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721818    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721820    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721823    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721825    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721830    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721833    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721836    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721839    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721842    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721845    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:11.725624 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721848    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.721853    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721965    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721970    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721973    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721976    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721979    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721982    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721985    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721988    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721991    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721994    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.721997    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722000    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722002    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:39:11.726033 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722005    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722008    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722010    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722013    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722016    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722019    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722022    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722024    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722026    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722029    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722031    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722034    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722036    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722039    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722042    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722045    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722047    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722050    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722052    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722055    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:39:11.726511 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722058    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722060    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722063    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722065    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722068    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722070    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722092    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722096    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722099    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722102    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722104    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722107    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722109    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722112    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722115    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722118    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722121    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722123    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722126    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722128    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:39:11.727025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722131    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722133    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722136    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722139    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722141    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722145    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722150    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722154    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722157    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722160    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722163    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722165    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722167    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722170    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722173    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722176    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722179    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722182    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722185    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:39:11.727503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722187    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722190    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722193    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722195    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722198    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722200    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722203    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722205    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722208    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722211    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722213    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722216    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722218    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722221    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722224    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722227    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722229    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722232    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722235    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:39:11.727997 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722237    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722240    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722242    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722245    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722248    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722250    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722253    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:11.722255    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.722260    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.722382    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.724337    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.725201    2575 server.go:1019] "Starting client certificate rotation"
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.725297    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:39:11.728490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.725968    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:39:11.728821 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.727571    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:39:11.731924 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.731907    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:39:11.744765 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.744736    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:39:11.747785 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.747768    2575 log.go:25] "Validated CRI v1 image API"
Sep 17 15:39:11.748637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.748617    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:39:11.750907 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.750889    2575 fs.go:135] Filesystem UUIDs: map[2fb91d0e-9636-4a35-88d4-b6ea220c1696:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7c3d8d89-95cb-4259-8bde-34727a3ef3e5:/dev/nvme0n1p4]
Sep 17 15:39:11.750972 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.750907    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:39:11.756155 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.756021    2575 manager.go:216] Machine: {Timestamp:2026-09-17 15:39:11.754574267 +0000 UTC m=+0.319611482 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097287 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2505ad73ced0bda094ab51723909f5 SystemUUID:ec2505ad-73ce-d0bd-a094-ab51723909f5 BootID:1d99eacd-deca-4e0f-8d12-cd780269c118 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:f6:a0:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:f6:a0:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:00:a8:1c:4f:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:39:11.756155 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.756149    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:39:11.756274 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.756238    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:39:11.758792 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.758773    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:39:11.759013 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.758988    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:39:11.759184 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.759019    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:39:11.760688 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.760484    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:39:11.760688 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.760498    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:39:11.760688 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.760507    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:39:11.766232 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.761110    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:39:11.766232 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.762243    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:39:11.767044 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767031    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:39:11.767201 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767190    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:39:11.767251 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767243    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:39:11.767280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767256    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:39:11.767280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767270    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:39:11.767280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767280    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:39:11.767396 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767289    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:39:11.767758 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767745    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:39:11.767813 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.767773    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:39:11.769969 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.769952    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:39:11.771734 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.771720    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:39:11.771772 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.771752    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:39:11.774143 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774130    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:39:11.774187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774150    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:39:11.774187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774157    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:39:11.774187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774163    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:39:11.774187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774171    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:39:11.774187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774177    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:39:11.774325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774193    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:39:11.774325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774199    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:39:11.774325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774205    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:39:11.774325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774210    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:39:11.774325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774219    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:39:11.774325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774224    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:39:11.774866 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774856    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:39:11.774866 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.774865    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:39:11.778866 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.778854    2575 server.go:1268] "Started kubelet"
Sep 17 15:39:11.778986 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.778932    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:39:11.779055 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.778981    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:39:11.779055 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.779026    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:39:11.779745 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.779725    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:39:11.779818 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.779732    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:39:11.779878 ip-10-0-133-29 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:39:11.780017 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.779867    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:39:11.783666 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.783641    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:39:11.785559 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.784490    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:39:11.786289 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.786258    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:39:11.788266 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.788242    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:39:11.788763 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.788738    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:39:11.789059 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.788997    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:11.789456 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.789438    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:39:11.789456 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.789438    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:39:11.789590 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.789464    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:39:11.789663 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.789647    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:39:11.789716 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.789664    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:39:11.791243 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791016    2575 factory.go:55] Registering systemd factory
Sep 17 15:39:11.791335 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791247    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:39:11.791510 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791495    2575 factory.go:153] Registering CRI-O factory
Sep 17 15:39:11.791565 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791514    2575 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:39:11.791611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791566    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:39:11.791611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791587    2575 factory.go:103] Registering Raw factory
Sep 17 15:39:11.791611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791604    2575 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:39:11.791748 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.791670    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:39:11.791748 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.791717    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:39:11.791959 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.791946    2575 manager.go:349] Starting recovery of all containers
Sep 17 15:39:11.792662 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.791740    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-29.ec2.internal.18d625e15ca96fd8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-29.ec2.internal,UID:ip-10-0-133-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-29.ec2.internal,},FirstTimestamp:2026-09-17 15:39:11.778828248 +0000 UTC m=+0.343865463,LastTimestamp:2026-09-17 15:39:11.778828248 +0000 UTC m=+0.343865463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-29.ec2.internal,}"
Sep 17 15:39:11.796281 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.796256    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlz8w"
Sep 17 15:39:11.802120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.802106    2575 manager.go:354] Recovery completed
Sep 17 15:39:11.804940 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.804925    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlz8w"
Sep 17 15:39:11.806204 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.806193    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:11.808301 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.808284    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:11.808370 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.808319    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:11.808370 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.808334    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:11.809348 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.809333    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:39:11.809348 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.809348    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:39:11.809475 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.809366    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:39:11.809475 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.809339    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:11.810199 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.810183    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:11.810264 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.810207    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:11.810264 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.810217    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:11.810353 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.810206    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-29.ec2.internal.18d625e15e6b2e8d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-29.ec2.internal,UID:ip-10-0-133-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-29.ec2.internal,},FirstTimestamp:2026-09-17 15:39:11.808302733 +0000 UTC m=+0.373339952,LastTimestamp:2026-09-17 15:39:11.808302733 +0000 UTC m=+0.373339952,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-29.ec2.internal,}"
Sep 17 15:39:11.811383 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.811370    2575 policy_none.go:49] "None policy: Start"
Sep 17 15:39:11.811383 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.811385    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:39:11.811479 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.811396    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:39:11.813333 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.813320    2575 policy_none.go:47] "Start"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.861632    2575 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.861672    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.861682    2575 server.go:87] "Starting device plugin registration server"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.862121    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.862137    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.862240    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.862331    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.862340    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.862882    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:39:11.864030 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.862926    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:11.915068 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.914942    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:39:11.916327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.916311    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:39:11.916394 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.916365    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:39:11.916394 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.916387    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:39:11.916483 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.916429    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:39:11.918778 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.918757    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:11.962920 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.962893    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:11.964157 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.964134    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:11.964278 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.964175    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:11.964278 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.964192    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:11.964278 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.964225    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-29.ec2.internal"
Sep 17 15:39:11.970822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:11.970803    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-29.ec2.internal"
Sep 17 15:39:11.970917 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.970827    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-29.ec2.internal\": node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:11.991287 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:11.991262    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.016820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.016787    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"]
Sep 17 15:39:12.016912 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.016861    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:12.018425 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.018410    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:12.018486 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.018441    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:12.018486 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.018452    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:12.020796 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.020777    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:12.020907 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.020890    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.020949 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.020937    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:12.022217 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.022201    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:12.022280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.022233    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:12.022280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.022243    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:12.022280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.022204    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:12.022381 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.022298    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:12.022381 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.022309    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:12.024549 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.024532    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.024628 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.024562    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:39:12.025400 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.025385    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:39:12.025491 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.025406    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:39:12.025491 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.025418    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:39:12.040697 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.040672    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-29.ec2.internal\" not found" node="ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.045183 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.045167    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-29.ec2.internal\" not found" node="ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.090838 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.090810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac07046bb35b604614256dce117e3f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.ec2.internal\" (UID: \"ac07046bb35b604614256dce117e3f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.090962 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.090843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89d2c05ee0794b868d5bce63b0952f0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal\" (UID: \"89d2c05ee0794b868d5bce63b0952f0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.090962 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.090868    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89d2c05ee0794b868d5bce63b0952f0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal\" (UID: \"89d2c05ee0794b868d5bce63b0952f0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.091830 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.091817    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.191435 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.191372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac07046bb35b604614256dce117e3f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.ec2.internal\" (UID: \"ac07046bb35b604614256dce117e3f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.191435 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.191406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89d2c05ee0794b868d5bce63b0952f0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal\" (UID: \"89d2c05ee0794b868d5bce63b0952f0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.191435 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.191422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89d2c05ee0794b868d5bce63b0952f0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal\" (UID: \"89d2c05ee0794b868d5bce63b0952f0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.191551 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.191486    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/89d2c05ee0794b868d5bce63b0952f0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal\" (UID: \"89d2c05ee0794b868d5bce63b0952f0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.191551 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.191491    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac07046bb35b604614256dce117e3f6a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-29.ec2.internal\" (UID: \"ac07046bb35b604614256dce117e3f6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.191551 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.191543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/89d2c05ee0794b868d5bce63b0952f0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal\" (UID: \"89d2c05ee0794b868d5bce63b0952f0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.192439 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.192424    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.293222 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.293194    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.343386 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.343354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.348196 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.348177    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.394182 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.394152    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.494682 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.494611    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.595069 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.595035    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.695535 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.695499    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.725692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.725667    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:39:12.726093 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.725820    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:39:12.796399 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:12.796370    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-29.ec2.internal\" not found"
Sep 17 15:39:12.807656 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.807597    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:12.810168 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.810121    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:34:12 +0000 UTC" deadline="2028-06-03 01:14:03.815013998 +0000 UTC"
Sep 17 15:39:12.810257 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.810170    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14985h34m51.004848767s"
Sep 17 15:39:12.889505 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.889471    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.902019 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.901996    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:39:12.903553 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.903539    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal"
Sep 17 15:39:12.913249 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.913233    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:39:12.948600 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:12.948564    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac07046bb35b604614256dce117e3f6a.slice/crio-6840654070c723f419f4130edfed857ef2f417b29d580b62a06499f14e40f0b2 WatchSource:0}: Error finding container 6840654070c723f419f4130edfed857ef2f417b29d580b62a06499f14e40f0b2: Status 404 returned error can't find the container with id 6840654070c723f419f4130edfed857ef2f417b29d580b62a06499f14e40f0b2
Sep 17 15:39:12.948863 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:12.948835    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d2c05ee0794b868d5bce63b0952f0a.slice/crio-30673ae4c577f08a8ea4c4359c9317d523a9151efb6506c4a0f85e90768fd1f8 WatchSource:0}: Error finding container 30673ae4c577f08a8ea4c4359c9317d523a9151efb6506c4a0f85e90768fd1f8: Status 404 returned error can't find the container with id 30673ae4c577f08a8ea4c4359c9317d523a9151efb6506c4a0f85e90768fd1f8
Sep 17 15:39:12.954052 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.954038    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:39:12.954881 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:12.954865    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:14.465734 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.465639    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:14.800710 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.800622    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:39:14.903582 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.903538    2575 apiserver.go:52] "Watching apiserver"
Sep 17 15:39:14.914272 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.914244    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:39:14.915531 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.915497    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-cpn9v","kube-system/konnectivity-agent-pwvjs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c","openshift-cluster-node-tuning-operator/tuned-dsszz","openshift-image-registry/node-ca-bl4h7","openshift-multus/multus-additional-cni-plugins-jj5tj","openshift-multus/multus-lxnls","openshift-network-operator/iptables-alerter-pdk2l","openshift-ovn-kubernetes/ovnkube-node-926t8","kube-system/global-pull-secret-syncer-4rljc","kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal","openshift-dns/node-resolver-7bkx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal","openshift-multus/network-metrics-daemon-wws4p"]
Sep 17 15:39:14.918003 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.917980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.920268 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.920245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:14.921170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.920905    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:39:14.921170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.920918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:39:14.921170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.920986    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.921170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.921138    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.921371 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.921227    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l2tjr\""
Sep 17 15:39:14.922566 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.922549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.923903 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.923886    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.924180 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.924161    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:39:14.924363 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.924349    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-28vfc\""
Sep 17 15:39:14.924600 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.924583    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.924812 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.924798    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:39:14.925534 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.925462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.925534 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.925494    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pxtfd\""
Sep 17 15:39:14.925693 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.925676    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.925966 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.925937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:39:14.926187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.926157    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.927058 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.927025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:14.928517 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.928501    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.929269 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.929248    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:14.929401 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.929384    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.929589 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.929569    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:39:14.929996 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.929682    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-znc77\""
Sep 17 15:39:14.930109 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.930091    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.930499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.930482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jvbjw\""
Sep 17 15:39:14.930616 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.930583    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.931668 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.931651    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:14.933816 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.933800    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:39:14.933966 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.933814    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:39:14.934044 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.933859    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m42g7\""
Sep 17 15:39:14.934617 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.934594    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:14.934700 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:14.934667    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:14.935759 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.935212    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:39:14.935759 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.935371    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.935759 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.935742    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2n42d\""
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936399    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936591    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-kubernetes\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysctl-d\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-system-cni-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936926    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysctl-conf\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-sys\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.936999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a82db29d-d227-44bd-9302-baa3e850e784-etc-tuned\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1408322c-4598-437b-9f34-af740c852bec-cni-binary-copy\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-hostroot\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aebc4895-93c5-402a-9073-f9f07b51538e-host-slash\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937115    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a82db29d-d227-44bd-9302-baa3e850e784-tmp\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c28ca2ab-73e6-447b-9eec-6a496d0f801c-host\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-socket-dir-parent\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-kubelet\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-lib-modules\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-var-lib-kubelet\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwjd\" (UniqueName: \"kubernetes.io/projected/a82db29d-d227-44bd-9302-baa3e850e784-kube-api-access-qtwjd\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-netns\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-multus-certs\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsk5q\" (UniqueName: \"kubernetes.io/projected/1408322c-4598-437b-9f34-af740c852bec-kube-api-access-xsk5q\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aebc4895-93c5-402a-9073-f9f07b51538e-iptables-alerter-script\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937579    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c28ca2ab-73e6-447b-9eec-6a496d0f801c-serviceca\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-k8s-cni-cncf-io\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-run\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937709    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kksb\" (UniqueName: \"kubernetes.io/projected/c28ca2ab-73e6-447b-9eec-6a496d0f801c-kube-api-access-4kksb\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:14.937851 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-cni-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-cnibin\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-conf-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysconfig\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.937991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-host\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-cni-bin\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1408322c-4598-437b-9f34-af740c852bec-multus-daemon-config\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-modprobe-d\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-systemd\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-cni-multus\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-etc-kubernetes\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bqwk\" (UniqueName: \"kubernetes.io/projected/aebc4895-93c5-402a-9073-f9f07b51538e-kube-api-access-9bqwk\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.938637 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.938614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzkr\" (UniqueName: \"kubernetes.io/projected/703e4747-9ef7-4344-98be-e8bf7e5c76f5-kube-api-access-rbzkr\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:14.939258 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.939219    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:39:14.939843 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.939792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:14.939925 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:14.939904    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:14.940533 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.940516    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:39:14.941185 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.941165    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:39:14.941346 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.941329    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s7ctq\""
Sep 17 15:39:14.941502 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.941417    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:39:14.943178 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.943138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:14.943259 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:14.943237    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:14.945940 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.945917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:14.949327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.949310    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:39:14.949799 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.949784    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:39:14.950283 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.950265    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pfrlt\""
Sep 17 15:39:14.969297 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.969273    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2v6pm"
Sep 17 15:39:14.980994 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:14.980932    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2v6pm"
Sep 17 15:39:15.025497 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.025472    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:39:15.039568 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-cni-multus\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.039568 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039605    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysctl-conf\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a82db29d-d227-44bd-9302-baa3e850e784-etc-tuned\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/391bbce6-2f5f-4c3d-9bef-2008106ef630-agent-certs\") pod \"konnectivity-agent-pwvjs\" (UID: \"391bbce6-2f5f-4c3d-9bef-2008106ef630\") " pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-run-netns\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039688    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-cni-multus\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-systemd\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039753    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9143008c-7b5a-47eb-82ab-ea5001a6d78a-tmp-dir\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.039802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039774    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f563bc-95bb-4b60-9067-b5252f2950bb-dbus\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-env-overrides\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039857    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-var-lib-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039902    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-device-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039906    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwjd\" (UniqueName: \"kubernetes.io/projected/a82db29d-d227-44bd-9302-baa3e850e784-kube-api-access-qtwjd\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-cni-binary-copy\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-sys-fs\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-log-socket\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.039858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysctl-conf\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aebc4895-93c5-402a-9073-f9f07b51538e-iptables-alerter-script\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c28ca2ab-73e6-447b-9eec-6a496d0f801c-serviceca\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.040186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/391bbce6-2f5f-4c3d-9bef-2008106ef630-konnectivity-ca\") pod \"konnectivity-agent-pwvjs\" (UID: \"391bbce6-2f5f-4c3d-9bef-2008106ef630\") " pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040195    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-socket-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-systemd-units\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-slash\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-ovn\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-ovnkube-config\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-k8s-cni-cncf-io\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-run\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-cnibin\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040436    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysconfig\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9143008c-7b5a-47eb-82ab-ea5001a6d78a-hosts-file\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040453    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040599    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c8321a-6335-4272-b2f3-a1919abbec16-ovn-node-metrics-cert\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffct\" (UniqueName: \"kubernetes.io/projected/f7c8321a-6335-4272-b2f3-a1919abbec16-kube-api-access-5ffct\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.040840 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1408322c-4598-437b-9f34-af740c852bec-multus-daemon-config\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040708    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-modprobe-d\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-systemd\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-run\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-cnibin\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-etc-kubernetes\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bqwk\" (UniqueName: \"kubernetes.io/projected/aebc4895-93c5-402a-9073-f9f07b51538e-kube-api-access-9bqwk\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzkr\" (UniqueName: \"kubernetes.io/projected/703e4747-9ef7-4344-98be-e8bf7e5c76f5-kube-api-access-rbzkr\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.040987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-kubernetes\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041023    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysctl-d\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-sys\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5hm\" (UniqueName: \"kubernetes.io/projected/9143008c-7b5a-47eb-82ab-ea5001a6d78a-kube-api-access-4z5hm\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-cni-bin\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-system-cni-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aebc4895-93c5-402a-9073-f9f07b51538e-iptables-alerter-script\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.041684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-run-ovn-kubernetes\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-cni-netd\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1408322c-4598-437b-9f34-af740c852bec-cni-binary-copy\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-hostroot\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aebc4895-93c5-402a-9073-f9f07b51538e-host-slash\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041320    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-system-cni-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a82db29d-d227-44bd-9302-baa3e850e784-tmp\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041359    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1408322c-4598-437b-9f34-af740c852bec-multus-daemon-config\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c28ca2ab-73e6-447b-9eec-6a496d0f801c-host\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysconfig\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-k8s-cni-cncf-io\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-etc-selinux\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-socket-dir-parent\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-kubelet\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-socket-dir-parent\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041528    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-lib-modules\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-var-lib-kubelet\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.042511 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041555    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-etc-kubernetes\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-systemd\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-etc-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-netns\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041601    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-kubernetes\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-sys\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-multus-certs\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-sysctl-d\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-hostroot\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c28ca2ab-73e6-447b-9eec-6a496d0f801c-host\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-etc-modprobe-d\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aebc4895-93c5-402a-9073-f9f07b51538e-host-slash\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsk5q\" (UniqueName: \"kubernetes.io/projected/1408322c-4598-437b-9f34-af740c852bec-kube-api-access-xsk5q\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-multus-certs\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kksb\" (UniqueName: \"kubernetes.io/projected/c28ca2ab-73e6-447b-9eec-6a496d0f801c-kube-api-access-4kksb\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-lib-modules\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-system-cni-dir\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.043350 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-cni-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-var-lib-kubelet\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-conf-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-conf-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.041892    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-kubelet\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042018    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-cnibin\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-multus-cni-dir\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-host\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042147    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-kubelet\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-node-log\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042201    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4v5\" (UniqueName: \"kubernetes.io/projected/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-kube-api-access-rw4v5\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/703e4747-9ef7-4344-98be-e8bf7e5c76f5-registration-dir\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-run-netns\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c28ca2ab-73e6-447b-9eec-6a496d0f801c-serviceca\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-cni-bin\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.044720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxgr\" (UniqueName: \"kubernetes.io/projected/19167536-b708-4f8b-b30e-b96491614264-kube-api-access-lcxgr\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.044720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f563bc-95bb-4b60-9067-b5252f2950bb-kubelet-config\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.044720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-ovnkube-script-lib\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.044720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1408322c-4598-437b-9f34-af740c852bec-host-var-lib-cni-bin\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a82db29d-d227-44bd-9302-baa3e850e784-host\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.044720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.042847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1408322c-4598-437b-9f34-af740c852bec-cni-binary-copy\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.044992 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.044965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a82db29d-d227-44bd-9302-baa3e850e784-etc-tuned\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.046047 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.046027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a82db29d-d227-44bd-9302-baa3e850e784-tmp\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.050483 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.050436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwjd\" (UniqueName: \"kubernetes.io/projected/a82db29d-d227-44bd-9302-baa3e850e784-kube-api-access-qtwjd\") pod \"tuned-dsszz\" (UID: \"a82db29d-d227-44bd-9302-baa3e850e784\") " pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.053343 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.053123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kksb\" (UniqueName: \"kubernetes.io/projected/c28ca2ab-73e6-447b-9eec-6a496d0f801c-kube-api-access-4kksb\") pod \"node-ca-bl4h7\" (UID: \"c28ca2ab-73e6-447b-9eec-6a496d0f801c\") " pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.053568 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.053540    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bqwk\" (UniqueName: \"kubernetes.io/projected/aebc4895-93c5-402a-9073-f9f07b51538e-kube-api-access-9bqwk\") pod \"iptables-alerter-pdk2l\" (UID: \"aebc4895-93c5-402a-9073-f9f07b51538e\") " pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.053822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.053804    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsk5q\" (UniqueName: \"kubernetes.io/projected/1408322c-4598-437b-9f34-af740c852bec-kube-api-access-xsk5q\") pod \"multus-lxnls\" (UID: \"1408322c-4598-437b-9f34-af740c852bec\") " pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.054609 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.054590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzkr\" (UniqueName: \"kubernetes.io/projected/703e4747-9ef7-4344-98be-e8bf7e5c76f5-kube-api-access-rbzkr\") pod \"aws-ebs-csi-driver-node-b4p7c\" (UID: \"703e4747-9ef7-4344-98be-e8bf7e5c76f5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.056343 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.056273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal" event={"ID":"ac07046bb35b604614256dce117e3f6a","Type":"ContainerStarted","Data":"6840654070c723f419f4130edfed857ef2f417b29d580b62a06499f14e40f0b2"}
Sep 17 15:39:15.057548 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.057528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal" event={"ID":"89d2c05ee0794b868d5bce63b0952f0a","Type":"ContainerStarted","Data":"30673ae4c577f08a8ea4c4359c9317d523a9151efb6506c4a0f85e90768fd1f8"}
Sep 17 15:39:15.143806 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143771    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-kubelet\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.143806 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-node-log\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4v5\" (UniqueName: \"kubernetes.io/projected/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-kube-api-access-rw4v5\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-node-log\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-kubelet\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.143990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxgr\" (UniqueName: \"kubernetes.io/projected/19167536-b708-4f8b-b30e-b96491614264-kube-api-access-lcxgr\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f563bc-95bb-4b60-9067-b5252f2950bb-kubelet-config\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.144041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-ovnkube-script-lib\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/391bbce6-2f5f-4c3d-9bef-2008106ef630-agent-certs\") pod \"konnectivity-agent-pwvjs\" (UID: \"391bbce6-2f5f-4c3d-9bef-2008106ef630\") " pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-run-netns\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-systemd\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f563bc-95bb-4b60-9067-b5252f2950bb-kubelet-config\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9143008c-7b5a-47eb-82ab-ea5001a6d78a-tmp-dir\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f563bc-95bb-4b60-9067-b5252f2950bb-dbus\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-env-overrides\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144272    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-var-lib-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144298    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-cni-binary-copy\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-log-socket\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/391bbce6-2f5f-4c3d-9bef-2008106ef630-konnectivity-ca\") pod \"konnectivity-agent-pwvjs\" (UID: \"391bbce6-2f5f-4c3d-9bef-2008106ef630\") " pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-systemd-units\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-slash\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-ovn\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-ovnkube-config\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9143008c-7b5a-47eb-82ab-ea5001a6d78a-hosts-file\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144581    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c8321a-6335-4272-b2f3-a1919abbec16-ovn-node-metrics-cert\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144574    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffct\" (UniqueName: \"kubernetes.io/projected/f7c8321a-6335-4272-b2f3-a1919abbec16-kube-api-access-5ffct\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-cnibin\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5hm\" (UniqueName: \"kubernetes.io/projected/9143008c-7b5a-47eb-82ab-ea5001a6d78a-kube-api-access-4z5hm\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-cni-bin\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.144822 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-run-ovn-kubernetes\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-ovnkube-script-lib\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-run-netns\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-cni-netd\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-cni-netd\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144895    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f563bc-95bb-4b60-9067-b5252f2950bb-dbus\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-etc-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-system-cni-dir\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.145061    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-etc-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-system-cni-dir\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.145149    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:39:15.645109736 +0000 UTC m=+3.075273517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9143008c-7b5a-47eb-82ab-ea5001a6d78a-tmp-dir\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-env-overrides\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9143008c-7b5a-47eb-82ab-ea5001a6d78a-hosts-file\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.145490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.144300    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-systemd\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-var-lib-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145688    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-log-socket\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145553    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145520    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-openvswitch\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145698    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-cni-binary-copy\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-slash\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.145813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-run-ovn\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.145721    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.145995    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:15.645971358 +0000 UTC m=+3.076135132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146048    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-cni-bin\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19167536-b708-4f8b-b30e-b96491614264-cnibin\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-host-run-ovn-kubernetes\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146175 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f7c8321a-6335-4272-b2f3-a1919abbec16-systemd-units\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.146712 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146223    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/391bbce6-2f5f-4c3d-9bef-2008106ef630-konnectivity-ca\") pod \"konnectivity-agent-pwvjs\" (UID: \"391bbce6-2f5f-4c3d-9bef-2008106ef630\") " pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.146712 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19167536-b708-4f8b-b30e-b96491614264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.146712 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.146338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f7c8321a-6335-4272-b2f3-a1919abbec16-ovnkube-config\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.148815 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.148790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c8321a-6335-4272-b2f3-a1919abbec16-ovn-node-metrics-cert\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.148922 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.148881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/391bbce6-2f5f-4c3d-9bef-2008106ef630-agent-certs\") pod \"konnectivity-agent-pwvjs\" (UID: \"391bbce6-2f5f-4c3d-9bef-2008106ef630\") " pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.157092 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.157070    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:15.157092 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.157095    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:15.157224 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.157109    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:15.157263 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.157228    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:15.657204984 +0000 UTC m=+3.087368760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:15.158425 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.158405    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4v5\" (UniqueName: \"kubernetes.io/projected/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-kube-api-access-rw4v5\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:15.159132 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.159113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffct\" (UniqueName: \"kubernetes.io/projected/f7c8321a-6335-4272-b2f3-a1919abbec16-kube-api-access-5ffct\") pod \"ovnkube-node-926t8\" (UID: \"f7c8321a-6335-4272-b2f3-a1919abbec16\") " pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.159385 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.159367    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5hm\" (UniqueName: \"kubernetes.io/projected/9143008c-7b5a-47eb-82ab-ea5001a6d78a-kube-api-access-4z5hm\") pod \"node-resolver-7bkx8\" (UID: \"9143008c-7b5a-47eb-82ab-ea5001a6d78a\") " pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.160550 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.160526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxgr\" (UniqueName: \"kubernetes.io/projected/19167536-b708-4f8b-b30e-b96491614264-kube-api-access-lcxgr\") pod \"multus-additional-cni-plugins-jj5tj\" (UID: \"19167536-b708-4f8b-b30e-b96491614264\") " pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.235073 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.235033    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxnls"
Sep 17 15:39:15.245674 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.245645    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdk2l"
Sep 17 15:39:15.254420 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.254386    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c"
Sep 17 15:39:15.259659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.259636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dsszz"
Sep 17 15:39:15.267381 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.267358    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bl4h7"
Sep 17 15:39:15.278996 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.278973    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jj5tj"
Sep 17 15:39:15.284683 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.284664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:15.290482 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.290463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:15.295165 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.295146    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7bkx8"
Sep 17 15:39:15.647937 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.647903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:15.648095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.647964    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:15.648095 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.648071    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:15.648095 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.648075    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:15.648218 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.648129    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:39:16.648114784 +0000 UTC m=+4.078278562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:15.648218 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.648144    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:16.648137984 +0000 UTC m=+4.078301743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:15.748962 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.748924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:15.749149 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.749108    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:15.749149 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.749130    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:15.749149 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.749139    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:15.749306 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:15.749209    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:16.74919462 +0000 UTC m=+4.179358379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:15.984783 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.984705    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:34:14 +0000 UTC" deadline="2028-05-24 12:36:31.959801689 +0000 UTC"
Sep 17 15:39:15.984783 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:15.984733    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14756h57m15.9750713s"
Sep 17 15:39:16.037128 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.037099    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebc4895_93c5_402a_9073_f9f07b51538e.slice/crio-2528f5d96fcf886a4e54d4924d3f460e587d9cbaec30ac0a0c595006fbd3d774 WatchSource:0}: Error finding container 2528f5d96fcf886a4e54d4924d3f460e587d9cbaec30ac0a0c595006fbd3d774: Status 404 returned error can't find the container with id 2528f5d96fcf886a4e54d4924d3f460e587d9cbaec30ac0a0c595006fbd3d774
Sep 17 15:39:16.042025 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.041998    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82db29d_d227_44bd_9302_baa3e850e784.slice/crio-8ed23131ea35dea9bf41c68b70f2e85f8940b16985cdf6d6634d637a9826567c WatchSource:0}: Error finding container 8ed23131ea35dea9bf41c68b70f2e85f8940b16985cdf6d6634d637a9826567c: Status 404 returned error can't find the container with id 8ed23131ea35dea9bf41c68b70f2e85f8940b16985cdf6d6634d637a9826567c
Sep 17 15:39:16.043155 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.043126    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1408322c_4598_437b_9f34_af740c852bec.slice/crio-12b1bebe06617c4b601709702e528d37550ec7808fde6df98a97cc466d6fb321 WatchSource:0}: Error finding container 12b1bebe06617c4b601709702e528d37550ec7808fde6df98a97cc466d6fb321: Status 404 returned error can't find the container with id 12b1bebe06617c4b601709702e528d37550ec7808fde6df98a97cc466d6fb321
Sep 17 15:39:16.044121 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.044102    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19167536_b708_4f8b_b30e_b96491614264.slice/crio-a728ddcf7d7594e8d7be868641c82df0c31a95892e6eecbc5f7a91125d9bfbe1 WatchSource:0}: Error finding container a728ddcf7d7594e8d7be868641c82df0c31a95892e6eecbc5f7a91125d9bfbe1: Status 404 returned error can't find the container with id a728ddcf7d7594e8d7be868641c82df0c31a95892e6eecbc5f7a91125d9bfbe1
Sep 17 15:39:16.044931 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.044885    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9143008c_7b5a_47eb_82ab_ea5001a6d78a.slice/crio-ea3d083f45cbcdeee9809053966d5696b19592457267e164a864c939067e11f4 WatchSource:0}: Error finding container ea3d083f45cbcdeee9809053966d5696b19592457267e164a864c939067e11f4: Status 404 returned error can't find the container with id ea3d083f45cbcdeee9809053966d5696b19592457267e164a864c939067e11f4
Sep 17 15:39:16.046887 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.046185    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c8321a_6335_4272_b2f3_a1919abbec16.slice/crio-31cd086b0fcd0beb4cc1226ebc3506061fd25c10e860cbba5aee3b88b8b30029 WatchSource:0}: Error finding container 31cd086b0fcd0beb4cc1226ebc3506061fd25c10e860cbba5aee3b88b8b30029: Status 404 returned error can't find the container with id 31cd086b0fcd0beb4cc1226ebc3506061fd25c10e860cbba5aee3b88b8b30029
Sep 17 15:39:16.052910 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.048165    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703e4747_9ef7_4344_98be_e8bf7e5c76f5.slice/crio-923ee94d1cd174b157e60842d77502a4de04c9514ff3cb5516779242284cb4b8 WatchSource:0}: Error finding container 923ee94d1cd174b157e60842d77502a4de04c9514ff3cb5516779242284cb4b8: Status 404 returned error can't find the container with id 923ee94d1cd174b157e60842d77502a4de04c9514ff3cb5516779242284cb4b8
Sep 17 15:39:16.052910 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:16.049034    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28ca2ab_73e6_447b_9eec_6a496d0f801c.slice/crio-ecd2e9e8ad5ff509bb751da95896f6c832f4a8a7ba9530d9d5c4ff6a804a7a51 WatchSource:0}: Error finding container ecd2e9e8ad5ff509bb751da95896f6c832f4a8a7ba9530d9d5c4ff6a804a7a51: Status 404 returned error can't find the container with id ecd2e9e8ad5ff509bb751da95896f6c832f4a8a7ba9530d9d5c4ff6a804a7a51
Sep 17 15:39:16.061132 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.061104    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pwvjs" event={"ID":"391bbce6-2f5f-4c3d-9bef-2008106ef630","Type":"ContainerStarted","Data":"eb1dcd17d498722b94ffb3d8cf1767148e6aeb0ac061d2a9ed63f0086b713e2d"}
Sep 17 15:39:16.062570 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.062535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c" event={"ID":"703e4747-9ef7-4344-98be-e8bf7e5c76f5","Type":"ContainerStarted","Data":"923ee94d1cd174b157e60842d77502a4de04c9514ff3cb5516779242284cb4b8"}
Sep 17 15:39:16.063939 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.063910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7bkx8" event={"ID":"9143008c-7b5a-47eb-82ab-ea5001a6d78a","Type":"ContainerStarted","Data":"ea3d083f45cbcdeee9809053966d5696b19592457267e164a864c939067e11f4"}
Sep 17 15:39:16.065061 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.065040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerStarted","Data":"a728ddcf7d7594e8d7be868641c82df0c31a95892e6eecbc5f7a91125d9bfbe1"}
Sep 17 15:39:16.066317 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.066293    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dsszz" event={"ID":"a82db29d-d227-44bd-9302-baa3e850e784","Type":"ContainerStarted","Data":"8ed23131ea35dea9bf41c68b70f2e85f8940b16985cdf6d6634d637a9826567c"}
Sep 17 15:39:16.067808 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.067787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdk2l" event={"ID":"aebc4895-93c5-402a-9073-f9f07b51538e","Type":"ContainerStarted","Data":"2528f5d96fcf886a4e54d4924d3f460e587d9cbaec30ac0a0c595006fbd3d774"}
Sep 17 15:39:16.068883 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.068861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bl4h7" event={"ID":"c28ca2ab-73e6-447b-9eec-6a496d0f801c","Type":"ContainerStarted","Data":"ecd2e9e8ad5ff509bb751da95896f6c832f4a8a7ba9530d9d5c4ff6a804a7a51"}
Sep 17 15:39:16.069923 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.069903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"31cd086b0fcd0beb4cc1226ebc3506061fd25c10e860cbba5aee3b88b8b30029"}
Sep 17 15:39:16.070899 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.070881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxnls" event={"ID":"1408322c-4598-437b-9f34-af740c852bec","Type":"ContainerStarted","Data":"12b1bebe06617c4b601709702e528d37550ec7808fde6df98a97cc466d6fb321"}
Sep 17 15:39:16.655961 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.655904    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:16.656138 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.655998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:16.656200 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.656143    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:16.656280 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.656264    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:39:18.65624446 +0000 UTC m=+6.086408224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:16.656686 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.656661    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:16.656777 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.656713    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:18.656696001 +0000 UTC m=+6.086859773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:16.757277 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.757240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:16.757437 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.757391    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:16.757437 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.757411    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:16.757437 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.757423    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:16.757596 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:16.757476    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:18.757459401 +0000 UTC m=+6.187623174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:16.985280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.985139    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:34:14 +0000 UTC" deadline="2028-03-08 08:37:37.864439864 +0000 UTC"
Sep 17 15:39:16.985280 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:16.985199    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12904h58m20.879245004s"
Sep 17 15:39:17.056247 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.056216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:17.056411 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:17.056343    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:17.056795 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.056772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:17.056916 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:17.056875    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:17.056995 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.056939    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:17.057055 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:17.057027    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:17.086790 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.086670    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal" event={"ID":"ac07046bb35b604614256dce117e3f6a","Type":"ContainerStarted","Data":"43f149b0f8ba6ba777e7c224ec8580074018446d44a9f2481eff0bd24bced4ea"}
Sep 17 15:39:17.093825 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.093349    2575 generic.go:356] "Generic (PLEG): container finished" podID="89d2c05ee0794b868d5bce63b0952f0a" containerID="84e2a08798b2a81111a84b1c69fdc445841d2e2e5053586d5cd84ed9e20dde84" exitCode=0
Sep 17 15:39:17.093825 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.093388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal" event={"ID":"89d2c05ee0794b868d5bce63b0952f0a","Type":"ContainerDied","Data":"84e2a08798b2a81111a84b1c69fdc445841d2e2e5053586d5cd84ed9e20dde84"}
Sep 17 15:39:17.124617 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:17.124538    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-29.ec2.internal" podStartSLOduration=3.124518986 podStartE2EDuration="3.124518986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:17.108317286 +0000 UTC m=+4.538481068" watchObservedRunningTime="2026-09-17 15:39:17.124518986 +0000 UTC m=+4.554682770"
Sep 17 15:39:18.130493 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:18.130415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal" event={"ID":"89d2c05ee0794b868d5bce63b0952f0a","Type":"ContainerStarted","Data":"040d1a363005799824f775ccd797d6f646cc8fceda010ca10dfc32f57d927be2"}
Sep 17 15:39:18.150868 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:18.150707    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-29.ec2.internal" podStartSLOduration=4.150687121 podStartE2EDuration="4.150687121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:18.149505396 +0000 UTC m=+5.579669178" watchObservedRunningTime="2026-09-17 15:39:18.150687121 +0000 UTC m=+5.580850905"
Sep 17 15:39:18.680939 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:18.680901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:18.681144 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:18.680986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:18.681204 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.681142    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:18.681259 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.681204    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:39:22.681186367 +0000 UTC m=+10.111350147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:18.681631 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.681584    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:18.681720 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.681635    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:22.681619927 +0000 UTC m=+10.111783700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:18.782269 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:18.782228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:18.782417 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.782370    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:18.782417 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.782389    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:18.782417 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.782402    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:18.782580 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:18.782452    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:22.782435762 +0000 UTC m=+10.212599521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:19.051756 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:19.051679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:19.051913 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:19.051823    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:19.052007 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:19.051933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:19.052075 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:19.052033    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:19.052127 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:19.052082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:19.052175 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:19.052150    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:21.054493 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:21.054446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:21.054493 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:21.054461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:21.054493 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:21.054442    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:21.054969 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:21.054567    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:21.054969 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:21.054613    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:21.054969 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:21.054685    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:22.713079 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:22.713044    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:22.713529 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:22.713101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:22.713529 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.713190    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:22.713529 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.713224    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:22.713529 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.713258    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:30.713238313 +0000 UTC m=+18.143402090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:22.713529 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.713279    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:39:30.713269373 +0000 UTC m=+18.143433136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:22.813997 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:22.813963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:22.814188 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.814120    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:22.814188 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.814145    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:22.814188 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.814158    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:22.814387 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:22.814207    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:30.81419349 +0000 UTC m=+18.244357252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:23.053288 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:23.052683    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:23.053288 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:23.052831    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:23.053288 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:23.053031    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:23.053288 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:23.053123    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:23.053288 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:23.053196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:23.053678 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:23.053291    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:25.051989 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:25.051848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:25.051989 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:25.051859    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:25.051989 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:25.051986    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:25.051989 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:25.051990    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:25.052554 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:25.052064    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:25.052554 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:25.052328    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:27.052648 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:27.052418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:27.053131 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:27.052426    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:27.053131 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:27.052755    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:27.053131 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:27.052470    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:27.053131 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:27.052861    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:27.053131 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:27.052910    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:29.052435 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:29.052403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:29.052877 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:29.052408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:29.052877 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:29.052539    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:29.052877 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:29.052418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:29.052877 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:29.052628    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:29.052877 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:29.052673    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:30.773074 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:30.773034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:30.773074 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:30.773079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:30.773532 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.773171    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:30.773532 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.773201    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:30.773532 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.773232    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:39:46.773214355 +0000 UTC m=+34.203378117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:30.773532 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.773261    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:46.773243511 +0000 UTC m=+34.203407274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:30.874338 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:30.874304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:30.874523 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.874455    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:30.874523 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.874477    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:30.874523 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.874486    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:30.874651 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:30.874538    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:46.874522829 +0000 UTC m=+34.304686590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:31.051830 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:31.051650    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:31.051830 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:31.051701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:31.051830 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:31.051654    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:31.051830 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:31.051793    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:31.052141 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:31.051937    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:31.052141 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:31.052028    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:33.052956 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:33.052929    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:33.053391 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:33.053042    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:33.053391 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:33.053057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:33.053391 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:33.053065    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:33.053391 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:33.053156    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:33.053391 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:33.053264    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:35.051764 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.051541    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:35.052676 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.051560    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:35.052676 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:35.051865    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:35.052676 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.051576    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:35.052676 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:35.051942    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:35.052676 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:35.052065    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:35.168709 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.168661    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bl4h7" event={"ID":"c28ca2ab-73e6-447b-9eec-6a496d0f801c","Type":"ContainerStarted","Data":"b531ae5900e7b0c10639c55ca312437c979da4fdf82e77b78f7cc5f0c2a55f12"}
Sep 17 15:39:35.171277 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.171256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"98467b0e693651fa5585df381d679ff296c0a003620693c1b12b192cb9a336fc"}
Sep 17 15:39:35.171373 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.171314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"cb8084dc329ce5d5247aad20aa67b4882ead34c794f0f8b0a72e15edde07e151"}
Sep 17 15:39:35.171373 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.171349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"80341c203977206bb4c71a7f143b6069441bc8457dcf655fef8dbe2ca5cc55a7"}
Sep 17 15:39:35.171462 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.171385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"469b8828d6cd9bc53b9d0646973b6bd0de7ca451789d458f1a6ff2ca6be9f352"}
Sep 17 15:39:35.171462 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.171413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"9af0257afcaa94a4056d4f2cf134d475f93a44c2b47ac44f5f15ad6f580f178c"}
Sep 17 15:39:35.171462 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.171449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"b4fe68d89ce9ce627f4d80d6ab5cc6eca93bfa5b0689380a32e0f52fd71d282b"}
Sep 17 15:39:35.172634 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.172614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxnls" event={"ID":"1408322c-4598-437b-9f34-af740c852bec","Type":"ContainerStarted","Data":"9337d5349167483d99a74d4150b94e2cf92d02248a7cbf9fb5af831428a87bfc"}
Sep 17 15:39:35.174051 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.174032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pwvjs" event={"ID":"391bbce6-2f5f-4c3d-9bef-2008106ef630","Type":"ContainerStarted","Data":"72e0d3a145ab82bafa436a4239076218e8e92f7100d9ce62f1b7905e32894ae6"}
Sep 17 15:39:35.175309 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.175287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c" event={"ID":"703e4747-9ef7-4344-98be-e8bf7e5c76f5","Type":"ContainerStarted","Data":"22c4450f714179634eef2bd761281a9eb904bad208be6181700ef99b584a03ab"}
Sep 17 15:39:35.176479 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.176457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7bkx8" event={"ID":"9143008c-7b5a-47eb-82ab-ea5001a6d78a","Type":"ContainerStarted","Data":"d59a42cfe44fc96e8f5345ed47725f980a9b93d84e58ffb4b7cd2d6d45f82c7d"}
Sep 17 15:39:35.177781 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.177760    2575 generic.go:356] "Generic (PLEG): container finished" podID="19167536-b708-4f8b-b30e-b96491614264" containerID="ad6456ba7d2097c564e0ccae0b073a2f1825b09df4add37e2608c6ae784262e1" exitCode=0
Sep 17 15:39:35.177854 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.177823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerDied","Data":"ad6456ba7d2097c564e0ccae0b073a2f1825b09df4add37e2608c6ae784262e1"}
Sep 17 15:39:35.179267 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.179249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dsszz" event={"ID":"a82db29d-d227-44bd-9302-baa3e850e784","Type":"ContainerStarted","Data":"e2aa647de505ffe1241256e3957959ffa413bc082159b1f60895c14ee202414e"}
Sep 17 15:39:35.184924 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.184892    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bl4h7" podStartSLOduration=4.131197393 podStartE2EDuration="22.184875926s" totalImagesPullingTime="18.053678533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:35.184774092 +0000 UTC m=+22.614937871" watchObservedRunningTime="2026-09-17 15:39:35.184875926 +0000 UTC m=+22.615039705"
Sep 17 15:39:35.206660 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.206575    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7bkx8" podStartSLOduration=4.148123411 podStartE2EDuration="22.206561315s" totalImagesPullingTime="18.058437904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:35.206317381 +0000 UTC m=+22.636481172" watchObservedRunningTime="2026-09-17 15:39:35.206561315 +0000 UTC m=+22.636725093"
Sep 17 15:39:35.227030 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.226987    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lxnls" podStartSLOduration=4.156357871 podStartE2EDuration="22.226972719s" totalImagesPullingTime="18.070614848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:35.226823126 +0000 UTC m=+22.656986904" watchObservedRunningTime="2026-09-17 15:39:35.226972719 +0000 UTC m=+22.657136494"
Sep 17 15:39:35.245442 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.245402    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dsszz" podStartSLOduration=4.183179623 podStartE2EDuration="22.245388919s" totalImagesPullingTime="18.062209296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:35.245260735 +0000 UTC m=+22.675424514" watchObservedRunningTime="2026-09-17 15:39:35.245388919 +0000 UTC m=+22.675552701"
Sep 17 15:39:35.624297 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:35.624136    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:39:36.011158 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.011063    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:39:35.624292918Z","UUID":"8f8598b8-baf0-46ed-b794-5c5f83e12270","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:39:36.012918 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.012890    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:39:36.013065 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.012928    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:39:36.185776 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.185740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c" event={"ID":"703e4747-9ef7-4344-98be-e8bf7e5c76f5","Type":"ContainerStarted","Data":"48c38378621e398ad98c9402e13c28996ec657064be7bbba6765131a7146f951"}
Sep 17 15:39:36.188935 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.188902    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdk2l" event={"ID":"aebc4895-93c5-402a-9073-f9f07b51538e","Type":"ContainerStarted","Data":"c70a10d5435054d5e1b0374fc7329d267d5cfdf659bbe1edaa867345ad3334f0"}
Sep 17 15:39:36.206709 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.206646    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pwvjs" podStartSLOduration=5.266664843 podStartE2EDuration="23.206630267s" totalImagesPullingTime="17.939965424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:35.290969556 +0000 UTC m=+22.721133332" watchObservedRunningTime="2026-09-17 15:39:36.206630267 +0000 UTC m=+23.636794047"
Sep 17 15:39:36.207181 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:36.207141    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pdk2l" podStartSLOduration=5.13995588 podStartE2EDuration="23.207130394s" totalImagesPullingTime="18.067174514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:36.206392939 +0000 UTC m=+23.636556720" watchObservedRunningTime="2026-09-17 15:39:36.207130394 +0000 UTC m=+23.637294175"
Sep 17 15:39:37.051909 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.051867    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:37.051909 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.051902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:37.052153 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:37.052046    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:37.052153 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:37.052100    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:37.052278 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.052151    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:37.052278 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:37.052248    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:37.194317 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.194277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"66482ce2dc0f8928047c6c152b4a3717a430e99924f376eb0e2f33069fd27f5b"}
Sep 17 15:39:37.197088 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.197058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c" event={"ID":"703e4747-9ef7-4344-98be-e8bf7e5c76f5","Type":"ContainerStarted","Data":"1f1824df3c413599ca67c5233fa9701dd80617840dcf1a8395ffac6dfdd76330"}
Sep 17 15:39:37.217676 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.217635    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b4p7c" podStartSLOduration=4.015743557 podStartE2EDuration="24.217615947s" totalImagesPullingTime="20.20187239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:37.217410324 +0000 UTC m=+24.647574126" watchObservedRunningTime="2026-09-17 15:39:37.217615947 +0000 UTC m=+24.647779726"
Sep 17 15:39:37.805596 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.805568    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:37.806190 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:37.806176    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:38.200000 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:38.199902    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:38.200381 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:38.200336    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pwvjs"
Sep 17 15:39:39.051485 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:39.051457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:39.051710 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:39.051457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:39.051710 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:39.051457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:39.051710 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:39.051700    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:39.051868 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:39.051751    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:39.051868 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:39.051770    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:40.205696 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.205456    2575 generic.go:356] "Generic (PLEG): container finished" podID="19167536-b708-4f8b-b30e-b96491614264" containerID="6d27ba82212ed592e0e4865df3d2a43a76aaf07009138722dd265160b079041f" exitCode=0
Sep 17 15:39:40.206157 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.205537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerDied","Data":"6d27ba82212ed592e0e4865df3d2a43a76aaf07009138722dd265160b079041f"}
Sep 17 15:39:40.209702 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.209678    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" event={"ID":"f7c8321a-6335-4272-b2f3-a1919abbec16","Type":"ContainerStarted","Data":"28c7ebec7ee70e1020936ab066ec77150e40cdbe151086f447ff02810e856838"}
Sep 17 15:39:40.209981 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.209963    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:40.210059 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.210014    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:40.210059 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.210058    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:40.224888 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.224865    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:40.225178 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.225161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:39:40.256409 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:40.256367    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-926t8" podStartSLOduration=9.145169226 podStartE2EDuration="27.256353768s" totalImagesPullingTime="18.111184542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:40.254792916 +0000 UTC m=+27.684956695" watchObservedRunningTime="2026-09-17 15:39:40.256353768 +0000 UTC m=+27.686517935"
Sep 17 15:39:41.051879 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.051852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:41.052065 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.051854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:41.052065 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:41.051984    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:41.052065 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.051995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:41.052220 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:41.052080    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:41.052220 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:41.052178    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:41.214038 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.214005    2575 generic.go:356] "Generic (PLEG): container finished" podID="19167536-b708-4f8b-b30e-b96491614264" containerID="4e4cc4000086f89cbc69e151d37313b2a66b878d75855d71b893c8456065e2ce" exitCode=0
Sep 17 15:39:41.214551 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.214091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerDied","Data":"4e4cc4000086f89cbc69e151d37313b2a66b878d75855d71b893c8456065e2ce"}
Sep 17 15:39:41.396520 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.396331    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wws4p"]
Sep 17 15:39:41.396623 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.396593    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:41.396715 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:41.396684    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:41.400574 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.400509    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cpn9v"]
Sep 17 15:39:41.400705 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.400592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:41.400705 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:41.400662    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:41.402384 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.402363    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4rljc"]
Sep 17 15:39:41.402490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:41.402453    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:41.402565 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:41.402543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:42.218834 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:42.218755    2575 generic.go:356] "Generic (PLEG): container finished" podID="19167536-b708-4f8b-b30e-b96491614264" containerID="bcb4384e94dfe125f065dfaa7bc269b70bedf83256f007448e27266df8708c3b" exitCode=0
Sep 17 15:39:42.219181 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:42.218837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerDied","Data":"bcb4384e94dfe125f065dfaa7bc269b70bedf83256f007448e27266df8708c3b"}
Sep 17 15:39:43.053157 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:43.053130    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:43.053332 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:43.053241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:43.053332 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:43.053260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:43.053449 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:43.053345    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:43.053696 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:43.053670    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:43.053782 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:43.053764    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:45.052123 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:45.052088    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:45.052123 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:45.052114    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:45.052684 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:45.052207    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpn9v" podUID="201035b8-6e31-42d8-9bf6-5c1265bdaf76"
Sep 17 15:39:45.052684 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:45.052269    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wws4p" podUID="2aa01c61-e6a5-47d6-8a03-da7e34a2cee4"
Sep 17 15:39:45.052684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:45.052338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:45.052684 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:45.052485    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4rljc" podUID="21f563bc-95bb-4b60-9067-b5252f2950bb"
Sep 17 15:39:46.709749 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.709728    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-29.ec2.internal" event="NodeReady"
Sep 17 15:39:46.710142 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.709829    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:39:46.761302 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.761280    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5rnnw"]
Sep 17 15:39:46.775988 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.775968    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8gs9v"]
Sep 17 15:39:46.776092 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.776082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.779047 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.779022    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:39:46.779147 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.779025    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxcdv\""
Sep 17 15:39:46.779147 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.779033    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:39:46.788425 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.788406    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rnnw"]
Sep 17 15:39:46.788498 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.788437    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gs9v"]
Sep 17 15:39:46.788498 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.788496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:46.790896 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.790880    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:39:46.791010 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.790916    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:39:46.791010 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.790884    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p86vs\""
Sep 17 15:39:46.791010 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.790990    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:39:46.793195 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.793179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:46.793272 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.793219    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:46.793311 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.793302    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:46.793351 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.793307    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:46.793351 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.793343    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:18.793331919 +0000 UTC m=+66.223495677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:46.793427 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.793355    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret podName:21f563bc-95bb-4b60-9067-b5252f2950bb nodeName:}" failed. No retries permitted until 2026-09-17 15:40:18.793349618 +0000 UTC m=+66.223513376 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret") pod "global-pull-secret-syncer-4rljc" (UID: "21f563bc-95bb-4b60-9067-b5252f2950bb") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:46.893621 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5682f4af-bd6e-48da-9086-f07848261bdf-cert\") pod \"ingress-canary-8gs9v\" (UID: \"5682f4af-bd6e-48da-9086-f07848261bdf\") " pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:46.893621 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snhx\" (UniqueName: \"kubernetes.io/projected/5682f4af-bd6e-48da-9086-f07848261bdf-kube-api-access-8snhx\") pod \"ingress-canary-8gs9v\" (UID: \"5682f4af-bd6e-48da-9086-f07848261bdf\") " pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:46.893749 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:46.893749 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-tmp-dir\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.893849 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-config-volume\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.893849 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893814    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.893932 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.893853    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m62r\" (UniqueName: \"kubernetes.io/projected/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-kube-api-access-7m62r\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.893932 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.893911    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:46.894034 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.893933    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:46.894034 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.893945    2575 projected.go:196] Error preparing data for projected volume kube-api-access-664zx for pod openshift-network-diagnostics/network-check-target-cpn9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:46.894034 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.894023    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx podName:201035b8-6e31-42d8-9bf6-5c1265bdaf76 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:18.894006545 +0000 UTC m=+66.324170322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-664zx" (UniqueName: "kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx") pod "network-check-target-cpn9v" (UID: "201035b8-6e31-42d8-9bf6-5c1265bdaf76") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:46.995137 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-tmp-dir\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.995227 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-config-volume\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.995227 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.995309 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m62r\" (UniqueName: \"kubernetes.io/projected/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-kube-api-access-7m62r\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.995309 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5682f4af-bd6e-48da-9086-f07848261bdf-cert\") pod \"ingress-canary-8gs9v\" (UID: \"5682f4af-bd6e-48da-9086-f07848261bdf\") " pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:46.995309 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snhx\" (UniqueName: \"kubernetes.io/projected/5682f4af-bd6e-48da-9086-f07848261bdf-kube-api-access-8snhx\") pod \"ingress-canary-8gs9v\" (UID: \"5682f4af-bd6e-48da-9086-f07848261bdf\") " pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:46.995447 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.995406    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:46.995494 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:46.995470    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:39:47.495450951 +0000 UTC m=+34.925614894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:46.995709 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-tmp-dir\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.995928 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.995904    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-config-volume\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:46.999217 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:46.999200    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5682f4af-bd6e-48da-9086-f07848261bdf-cert\") pod \"ingress-canary-8gs9v\" (UID: \"5682f4af-bd6e-48da-9086-f07848261bdf\") " pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:47.006964 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.006927    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m62r\" (UniqueName: \"kubernetes.io/projected/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-kube-api-access-7m62r\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:47.007063 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.007021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snhx\" (UniqueName: \"kubernetes.io/projected/5682f4af-bd6e-48da-9086-f07848261bdf-kube-api-access-8snhx\") pod \"ingress-canary-8gs9v\" (UID: \"5682f4af-bd6e-48da-9086-f07848261bdf\") " pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:47.052363 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.052332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:39:47.052523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.052332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:39:47.052523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.052463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:39:47.055296 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.055144    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:47.055296 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.055149    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:47.055296 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.055195    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:47.055296 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.055207    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nm7x2\""
Sep 17 15:39:47.055633 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.055465    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm2sj\""
Sep 17 15:39:47.055633 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.055470    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:47.096497 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.096477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gs9v"
Sep 17 15:39:47.232906 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.232872    2575 generic.go:356] "Generic (PLEG): container finished" podID="19167536-b708-4f8b-b30e-b96491614264" containerID="1a0a2f4544cd4b45023da4c5005082a1da07e7b811a5a05c0a861c2a2c133155" exitCode=0
Sep 17 15:39:47.233051 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.232927    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerDied","Data":"1a0a2f4544cd4b45023da4c5005082a1da07e7b811a5a05c0a861c2a2c133155"}
Sep 17 15:39:47.404129 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.404095    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gs9v"]
Sep 17 15:39:47.442735 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:39:47.442668    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5682f4af_bd6e_48da_9086_f07848261bdf.slice/crio-140cd2196f4fd277958fd25242d833c70bacd47b5a7dbfddf3c7afc7dac92b78 WatchSource:0}: Error finding container 140cd2196f4fd277958fd25242d833c70bacd47b5a7dbfddf3c7afc7dac92b78: Status 404 returned error can't find the container with id 140cd2196f4fd277958fd25242d833c70bacd47b5a7dbfddf3c7afc7dac92b78
Sep 17 15:39:47.500223 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:47.500197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:47.500340 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:47.500323    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:47.500402 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:47.500390    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:39:48.500372007 +0000 UTC m=+35.930535781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:48.236609 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:48.236567    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gs9v" event={"ID":"5682f4af-bd6e-48da-9086-f07848261bdf","Type":"ContainerStarted","Data":"140cd2196f4fd277958fd25242d833c70bacd47b5a7dbfddf3c7afc7dac92b78"}
Sep 17 15:39:48.241416 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:48.241006    2575 generic.go:356] "Generic (PLEG): container finished" podID="19167536-b708-4f8b-b30e-b96491614264" containerID="334081635e7004c51b7315c989fbe8ad4d73d654f19a4176396c0011a7a865b7" exitCode=0
Sep 17 15:39:48.241416 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:48.241050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerDied","Data":"334081635e7004c51b7315c989fbe8ad4d73d654f19a4176396c0011a7a865b7"}
Sep 17 15:39:48.507472 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:48.507399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:48.507655 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:48.507549    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:48.507655 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:48.507627    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:39:50.507612541 +0000 UTC m=+37.937776299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:49.247725 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:49.247690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" event={"ID":"19167536-b708-4f8b-b30e-b96491614264","Type":"ContainerStarted","Data":"db617a597b021e076302b3ca2543deddbaf2febb0ded0c42019d309f062d0b23"}
Sep 17 15:39:49.272926 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:49.272876    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jj5tj" podStartSLOduration=8.03433631 podStartE2EDuration="36.272859298s" totalImagesPullingTime="28.238522988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:49.271701157 +0000 UTC m=+36.701864937" watchObservedRunningTime="2026-09-17 15:39:49.272859298 +0000 UTC m=+36.703023130"
Sep 17 15:39:50.251486 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:50.251453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gs9v" event={"ID":"5682f4af-bd6e-48da-9086-f07848261bdf","Type":"ContainerStarted","Data":"bef4848a181adec3f060ebed96ee271cd3317aad0d7c55d336ed0ebb8beeb39b"}
Sep 17 15:39:50.285941 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:50.285892    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8gs9v" podStartSLOduration=2.028379307 podStartE2EDuration="4.285879575s" totalImagesPullingTime="2.257500268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:50.285037168 +0000 UTC m=+37.715200947" watchObservedRunningTime="2026-09-17 15:39:50.285879575 +0000 UTC m=+37.716043352"
Sep 17 15:39:50.521609 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:50.521545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:50.521734 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:50.521666    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:50.521734 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:50.521730    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:39:54.521715893 +0000 UTC m=+41.951879652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:54.546596 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:39:54.546564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:39:54.546935 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:54.546706    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:54.546935 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:39:54.546765    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:40:02.546750941 +0000 UTC m=+49.976914699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:02.600556 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:02.600514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:40:02.600999 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:02.600656    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:40:02.600999 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:02.600711    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:40:18.600697139 +0000 UTC m=+66.030860897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:12.229823 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:12.229786    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926t8"
Sep 17 15:40:16.072774 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.072738    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-gvzhp"]
Sep 17 15:40:16.075167 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.075151    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.077589 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.077560    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 15:40:16.077707 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.077619    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2bx5t\""
Sep 17 15:40:16.078535 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.078514    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:40:16.078636 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.078556    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:40:16.078699 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.078646    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 15:40:16.083116 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.083099    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 15:40:16.087316 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.087296    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gvzhp"]
Sep 17 15:40:16.089811 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.089791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed0dd50-7be8-49e8-a57e-14e04452e87f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.089885 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.089827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ed0dd50-7be8-49e8-a57e-14e04452e87f-tmp\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.089885 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.089845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed0dd50-7be8-49e8-a57e-14e04452e87f-service-ca-bundle\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.089885 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.089863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5ed0dd50-7be8-49e8-a57e-14e04452e87f-snapshots\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.090009 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.089970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ed0dd50-7be8-49e8-a57e-14e04452e87f-serving-cert\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.090043 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.090017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d29t\" (UniqueName: \"kubernetes.io/projected/5ed0dd50-7be8-49e8-a57e-14e04452e87f-kube-api-access-6d29t\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190261 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ed0dd50-7be8-49e8-a57e-14e04452e87f-serving-cert\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190261 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d29t\" (UniqueName: \"kubernetes.io/projected/5ed0dd50-7be8-49e8-a57e-14e04452e87f-kube-api-access-6d29t\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190526 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed0dd50-7be8-49e8-a57e-14e04452e87f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190526 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ed0dd50-7be8-49e8-a57e-14e04452e87f-tmp\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190526 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed0dd50-7be8-49e8-a57e-14e04452e87f-service-ca-bundle\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190526 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5ed0dd50-7be8-49e8-a57e-14e04452e87f-snapshots\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.190786 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ed0dd50-7be8-49e8-a57e-14e04452e87f-tmp\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.191018 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.190997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed0dd50-7be8-49e8-a57e-14e04452e87f-service-ca-bundle\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.191090 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.191004    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5ed0dd50-7be8-49e8-a57e-14e04452e87f-snapshots\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.191192 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.191171    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ed0dd50-7be8-49e8-a57e-14e04452e87f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.193740 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.193724    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ed0dd50-7be8-49e8-a57e-14e04452e87f-serving-cert\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.199441 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.199418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d29t\" (UniqueName: \"kubernetes.io/projected/5ed0dd50-7be8-49e8-a57e-14e04452e87f-kube-api-access-6d29t\") pod \"insights-operator-5844b856db-gvzhp\" (UID: \"5ed0dd50-7be8-49e8-a57e-14e04452e87f\") " pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.384852 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.384767    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gvzhp"
Sep 17 15:40:16.654990 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:16.654897    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gvzhp"]
Sep 17 15:40:16.657984 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:16.657944    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed0dd50_7be8_49e8_a57e_14e04452e87f.slice/crio-e41b438d3e4674f15bd134a029da8f52a192e864fe6486608d5447eddef6c5ba WatchSource:0}: Error finding container e41b438d3e4674f15bd134a029da8f52a192e864fe6486608d5447eddef6c5ba: Status 404 returned error can't find the container with id e41b438d3e4674f15bd134a029da8f52a192e864fe6486608d5447eddef6c5ba
Sep 17 15:40:17.313100 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:17.313059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gvzhp" event={"ID":"5ed0dd50-7be8-49e8-a57e-14e04452e87f","Type":"ContainerStarted","Data":"e41b438d3e4674f15bd134a029da8f52a192e864fe6486608d5447eddef6c5ba"}
Sep 17 15:40:18.607898 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.607863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:40:18.608284 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:18.608027    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:40:18.608284 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:18.608094    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls podName:752d9f57-ac3c-4115-9b3a-ceb4c4fb58de nodeName:}" failed. No retries permitted until 2026-09-17 15:40:50.608078098 +0000 UTC m=+98.038241861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls") pod "dns-default-5rnnw" (UID: "752d9f57-ac3c-4115-9b3a-ceb4c4fb58de") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:18.809115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.809033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:40:18.809115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.809096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:40:18.811659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.811640    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:40:18.811792 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.811774    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:40:18.819517 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:18.819498    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:40:18.819606 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:18.819567    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs podName:2aa01c61-e6a5-47d6-8a03-da7e34a2cee4 nodeName:}" failed. No retries permitted until 2026-09-17 15:41:22.819540998 +0000 UTC m=+130.249704757 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs") pod "network-metrics-daemon-wws4p" (UID: "2aa01c61-e6a5-47d6-8a03-da7e34a2cee4") : secret "metrics-daemon-secret" not found
Sep 17 15:40:18.821845 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.821829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f563bc-95bb-4b60-9067-b5252f2950bb-original-pull-secret\") pod \"global-pull-secret-syncer-4rljc\" (UID: \"21f563bc-95bb-4b60-9067-b5252f2950bb\") " pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:40:18.873601 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.873575    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4rljc"
Sep 17 15:40:18.910292 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.910257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:40:18.912895 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.912876    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:40:18.923991 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.923963    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:40:18.934741 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:18.934719    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664zx\" (UniqueName: \"kubernetes.io/projected/201035b8-6e31-42d8-9bf6-5c1265bdaf76-kube-api-access-664zx\") pod \"network-check-target-cpn9v\" (UID: \"201035b8-6e31-42d8-9bf6-5c1265bdaf76\") " pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:40:19.157814 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.157727    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4rljc"]
Sep 17 15:40:19.160651 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:19.160617    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f563bc_95bb_4b60_9067_b5252f2950bb.slice/crio-7016ba168b9330240151411b5bda2c27d7141a2396605fa66f4e449209cd940c WatchSource:0}: Error finding container 7016ba168b9330240151411b5bda2c27d7141a2396605fa66f4e449209cd940c: Status 404 returned error can't find the container with id 7016ba168b9330240151411b5bda2c27d7141a2396605fa66f4e449209cd940c
Sep 17 15:40:19.181999 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.181977    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nm7x2\""
Sep 17 15:40:19.189815 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.189798    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:40:19.318494 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.318462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4rljc" event={"ID":"21f563bc-95bb-4b60-9067-b5252f2950bb","Type":"ContainerStarted","Data":"7016ba168b9330240151411b5bda2c27d7141a2396605fa66f4e449209cd940c"}
Sep 17 15:40:19.320051 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.320020    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gvzhp" event={"ID":"5ed0dd50-7be8-49e8-a57e-14e04452e87f","Type":"ContainerStarted","Data":"e04680a7e17aa7f708ca1891910ba7aab87a40efcc0ac5dd2e4a88da42dca8a6"}
Sep 17 15:40:19.340114 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.340070    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-gvzhp" podStartSLOduration=1.49603906 podStartE2EDuration="3.340055567s" totalImagesPullingTime="1.844016507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:19.339258348 +0000 UTC m=+66.769422135" watchObservedRunningTime="2026-09-17 15:40:19.340055567 +0000 UTC m=+66.770219348"
Sep 17 15:40:19.456811 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.456731    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cpn9v"]
Sep 17 15:40:19.460160 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:19.460139    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201035b8_6e31_42d8_9bf6_5c1265bdaf76.slice/crio-bb513d138b29cdedc8ff189b6211b7087524c8de6cf5c3a390a4282d7f2594f9 WatchSource:0}: Error finding container bb513d138b29cdedc8ff189b6211b7087524c8de6cf5c3a390a4282d7f2594f9: Status 404 returned error can't find the container with id bb513d138b29cdedc8ff189b6211b7087524c8de6cf5c3a390a4282d7f2594f9
Sep 17 15:40:19.658778 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.658744    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mrj2w"]
Sep 17 15:40:19.661141 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.661122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.663845 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.663825    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pdt22\""
Sep 17 15:40:19.664469 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.664452    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:40:19.664573 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.664535    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:40:19.682624 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.682597    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mrj2w"]
Sep 17 15:40:19.716662 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.716604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.716662 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.716637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-data-volume\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.716854 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.716711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-crio-socket\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.716854 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.716738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-api-access-pwx75\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.716854 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.716766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818011 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.817976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-crio-socket\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818168 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.818022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-api-access-pwx75\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818168 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.818061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818168 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.818105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-crio-socket\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818168 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.818123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818168 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.818154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-data-volume\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818376 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:19.818186    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:19.818376 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:19.818248    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.318229979 +0000 UTC m=+67.748393740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:19.818496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.818475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-data-volume\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:19.818578 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:19.818565    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:19.818624 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:19.818615    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.31860067 +0000 UTC m=+67.748764442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:19.829254 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:19.829228    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwx75\" (UniqueName: \"kubernetes.io/projected/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-api-access-pwx75\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:20.322155 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:20.321621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:20.322155 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:20.321697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:20.322155 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:20.321851    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.322155 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:20.321910    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.32189091 +0000 UTC m=+68.752054682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.322155 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:20.322077    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.322155 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:20.322133    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.32211747 +0000 UTC m=+68.752281246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.324935 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:20.324904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cpn9v" event={"ID":"201035b8-6e31-42d8-9bf6-5c1265bdaf76","Type":"ContainerStarted","Data":"bb513d138b29cdedc8ff189b6211b7087524c8de6cf5c3a390a4282d7f2594f9"}
Sep 17 15:40:21.013421 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.013354    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"]
Sep 17 15:40:21.015869 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.015849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.018559 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.018410    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:21.018559 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.018501    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 15:40:21.018745 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.018716    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4z6lj\""
Sep 17 15:40:21.019738 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.019719    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:21.027221 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.027197    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"]
Sep 17 15:40:21.027499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.027469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g5fk\" (UniqueName: \"kubernetes.io/projected/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-kube-api-access-8g5fk\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.027613 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.027512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.128414 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.128380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g5fk\" (UniqueName: \"kubernetes.io/projected/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-kube-api-access-8g5fk\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.128584 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.128430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.128584 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.128571    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:21.128704 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.128632    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls podName:9ffecac2-0ff5-4c1d-b6a3-c6c26640664b nodeName:}" failed. No retries permitted until 2026-09-17 15:40:21.628614135 +0000 UTC m=+69.058777898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c5kf6" (UID: "9ffecac2-0ff5-4c1d-b6a3-c6c26640664b") : secret "samples-operator-tls" not found
Sep 17 15:40:21.143691 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.143664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g5fk\" (UniqueName: \"kubernetes.io/projected/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-kube-api-access-8g5fk\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.330228 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.330150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:21.330228 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.330222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:21.330408 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.330274    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:21.330408 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.330339    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:21.330408 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.330382    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.330364149 +0000 UTC m=+70.760527925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:21.330408 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.330399    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.330390577 +0000 UTC m=+70.760554336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:21.357386 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.357360    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7bkx8_9143008c-7b5a-47eb-82ab-ea5001a6d78a/dns-node-resolver/0.log"
Sep 17 15:40:21.632298 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:21.632216    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:21.632420 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.632342    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:21.632420 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:21.632399    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls podName:9ffecac2-0ff5-4c1d-b6a3-c6c26640664b nodeName:}" failed. No retries permitted until 2026-09-17 15:40:22.632386483 +0000 UTC m=+70.062550242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c5kf6" (UID: "9ffecac2-0ff5-4c1d-b6a3-c6c26640664b") : secret "samples-operator-tls" not found
Sep 17 15:40:22.354368 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:22.354338    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bl4h7_c28ca2ab-73e6-447b-9eec-6a496d0f801c/node-ca/0.log"
Sep 17 15:40:22.638989 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:22.638884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:22.639135 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:22.639079    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:22.639195 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:22.639160    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls podName:9ffecac2-0ff5-4c1d-b6a3-c6c26640664b nodeName:}" failed. No retries permitted until 2026-09-17 15:40:24.639141773 +0000 UTC m=+72.069305533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c5kf6" (UID: "9ffecac2-0ff5-4c1d-b6a3-c6c26640664b") : secret "samples-operator-tls" not found
Sep 17 15:40:22.754115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:22.754083    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gs9v_5682f4af-bd6e-48da-9086-f07848261bdf/serve-healthcheck-canary/0.log"
Sep 17 15:40:23.346098 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:23.346062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:23.346263 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:23.346155    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:23.346263 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:23.346196    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:23.346374 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:23.346264    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:27.34624702 +0000 UTC m=+74.776410799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:23.346374 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:23.346266    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:23.346374 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:23.346321    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:27.346306966 +0000 UTC m=+74.776470726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:24.337699 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:24.337665    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4rljc" event={"ID":"21f563bc-95bb-4b60-9067-b5252f2950bb","Type":"ContainerStarted","Data":"9378ba0a13e2aaae513ce3c2aba7d1f98412461926953562e58e149cdf41484e"}
Sep 17 15:40:24.339162 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:24.339139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cpn9v" event={"ID":"201035b8-6e31-42d8-9bf6-5c1265bdaf76","Type":"ContainerStarted","Data":"7ba218dc6ef6e682891b668ebc54a49fad33a6f3ade442b4bc6a5809349cdc2d"}
Sep 17 15:40:24.339287 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:24.339271    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:40:24.354397 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:24.354355    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4rljc" podStartSLOduration=66.810696392 podStartE2EDuration="1m11.3543364s" totalImagesPullingTime="4.543640008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:24.353538491 +0000 UTC m=+71.783702263" watchObservedRunningTime="2026-09-17 15:40:24.3543364 +0000 UTC m=+71.784500181"
Sep 17 15:40:24.371187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:24.371112    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cpn9v" podStartSLOduration=67.130416886 podStartE2EDuration="1m11.371098879s" totalImagesPullingTime="4.240681993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:24.370205895 +0000 UTC m=+71.800369687" watchObservedRunningTime="2026-09-17 15:40:24.371098879 +0000 UTC m=+71.801262651"
Sep 17 15:40:24.656622 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:24.656513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:24.656766 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:24.656640    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:24.656766 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:24.656695    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls podName:9ffecac2-0ff5-4c1d-b6a3-c6c26640664b nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.656680658 +0000 UTC m=+76.086844417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c5kf6" (UID: "9ffecac2-0ff5-4c1d-b6a3-c6c26640664b") : secret "samples-operator-tls" not found
Sep 17 15:40:26.124142 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.124104    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"]
Sep 17 15:40:26.126158 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.126137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.128613 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.128578    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:26.128613 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.128589    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qx5cn\""
Sep 17 15:40:26.128782 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.128619    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 15:40:26.129510 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.129491    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 15:40:26.129581 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.129499    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 15:40:26.137582 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.137558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"]
Sep 17 15:40:26.168110 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.168070    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.168289 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.168127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.168289 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.168172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzj9\" (UniqueName: \"kubernetes.io/projected/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-kube-api-access-kfzj9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.269071 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.269027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.269181 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.269095    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.269181 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.269135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzj9\" (UniqueName: \"kubernetes.io/projected/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-kube-api-access-kfzj9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.269595 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.269574    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.271464 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.271444    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.278935 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.278908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzj9\" (UniqueName: \"kubernetes.io/projected/7bf6160d-e4bf-4e3c-9884-59ecab0f839d-kube-api-access-kfzj9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n44s8\" (UID: \"7bf6160d-e4bf-4e3c-9884-59ecab0f839d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.434755 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.434668    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"
Sep 17 15:40:26.703662 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:26.703579    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8"]
Sep 17 15:40:26.706289 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:26.706262    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf6160d_e4bf_4e3c_9884_59ecab0f839d.slice/crio-9ab99a825f91fbc23773577d2adba2a18701c152f432d6c4c07c7877b8f4b45e WatchSource:0}: Error finding container 9ab99a825f91fbc23773577d2adba2a18701c152f432d6c4c07c7877b8f4b45e: Status 404 returned error can't find the container with id 9ab99a825f91fbc23773577d2adba2a18701c152f432d6c4c07c7877b8f4b45e
Sep 17 15:40:27.286996 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.286941    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"]
Sep 17 15:40:27.289020 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.288991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"
Sep 17 15:40:27.291465 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.291447    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2qgcj\""
Sep 17 15:40:27.298471 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.298452    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"]
Sep 17 15:40:27.348033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.347988    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8" event={"ID":"7bf6160d-e4bf-4e3c-9884-59ecab0f839d","Type":"ContainerStarted","Data":"9ab99a825f91fbc23773577d2adba2a18701c152f432d6c4c07c7877b8f4b45e"}
Sep 17 15:40:27.379734 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.379689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:27.379898 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.379783    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhcf\" (UniqueName: \"kubernetes.io/projected/a9df70ce-5f32-4e2c-88e1-520f5523d090-kube-api-access-knhcf\") pod \"network-check-source-6f8b768455-s57pz\" (UID: \"a9df70ce-5f32-4e2c-88e1-520f5523d090\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"
Sep 17 15:40:27.379898 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:27.379838    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:27.379898 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:27.379895    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:35.379876722 +0000 UTC m=+82.810040482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:27.380109 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.379913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:27.380109 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:27.380034    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:27.380109 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:27.380082    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:35.380069446 +0000 UTC m=+82.810233208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:27.480908 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.480865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhcf\" (UniqueName: \"kubernetes.io/projected/a9df70ce-5f32-4e2c-88e1-520f5523d090-kube-api-access-knhcf\") pod \"network-check-source-6f8b768455-s57pz\" (UID: \"a9df70ce-5f32-4e2c-88e1-520f5523d090\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"
Sep 17 15:40:27.491379 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.491340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhcf\" (UniqueName: \"kubernetes.io/projected/a9df70ce-5f32-4e2c-88e1-520f5523d090-kube-api-access-knhcf\") pod \"network-check-source-6f8b768455-s57pz\" (UID: \"a9df70ce-5f32-4e2c-88e1-520f5523d090\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"
Sep 17 15:40:27.597697 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.597607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"
Sep 17 15:40:27.876466 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:27.876386    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-s57pz"]
Sep 17 15:40:27.880475 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:27.880447    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9df70ce_5f32_4e2c_88e1_520f5523d090.slice/crio-c9875d127aa1a2f2c379e9c9ca65587a6a3e398c898ef9becc63711d087f9cda WatchSource:0}: Error finding container c9875d127aa1a2f2c379e9c9ca65587a6a3e398c898ef9becc63711d087f9cda: Status 404 returned error can't find the container with id c9875d127aa1a2f2c379e9c9ca65587a6a3e398c898ef9becc63711d087f9cda
Sep 17 15:40:28.352301 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:28.352264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz" event={"ID":"a9df70ce-5f32-4e2c-88e1-520f5523d090","Type":"ContainerStarted","Data":"cfeb96f39407cb876bee458b5c9931465cb65dfc56d15545ba9bc2e9ab523812"}
Sep 17 15:40:28.352699 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:28.352322    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz" event={"ID":"a9df70ce-5f32-4e2c-88e1-520f5523d090","Type":"ContainerStarted","Data":"c9875d127aa1a2f2c379e9c9ca65587a6a3e398c898ef9becc63711d087f9cda"}
Sep 17 15:40:28.370420 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:28.370362    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-s57pz" podStartSLOduration=1.3703424069999999 podStartE2EDuration="1.370342407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:28.369473796 +0000 UTC m=+75.799637577" watchObservedRunningTime="2026-09-17 15:40:28.370342407 +0000 UTC m=+75.800506169"
Sep 17 15:40:28.689523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:28.689481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:28.689674 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:28.689619    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:40:28.689674 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:28.689673    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls podName:9ffecac2-0ff5-4c1d-b6a3-c6c26640664b nodeName:}" failed. No retries permitted until 2026-09-17 15:40:36.689659398 +0000 UTC m=+84.119823158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-c5kf6" (UID: "9ffecac2-0ff5-4c1d-b6a3-c6c26640664b") : secret "samples-operator-tls" not found
Sep 17 15:40:29.358780 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:29.358745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8" event={"ID":"7bf6160d-e4bf-4e3c-9884-59ecab0f839d","Type":"ContainerStarted","Data":"d690a78291eedab2704b4ad70b9aa6bffdf6304cd8bffae7acfb0c3221b8317f"}
Sep 17 15:40:31.835931 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.835878    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8" podStartSLOduration=3.981862333 podStartE2EDuration="5.835861942s" totalImagesPullingTime="1.853999609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:29.383423563 +0000 UTC m=+76.813587545" watchObservedRunningTime="2026-09-17 15:40:31.835861942 +0000 UTC m=+79.266025722"
Sep 17 15:40:31.836314 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.836046    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-b6vfj"]
Sep 17 15:40:31.838186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.838172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:31.841293 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.841272    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 15:40:31.842275 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.842256    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 15:40:31.842365 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.842259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 15:40:31.842365 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.842263    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 15:40:31.842365 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.842310    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ct24t\""
Sep 17 15:40:31.849473 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.849450    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-b6vfj"]
Sep 17 15:40:31.913150 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.913119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cbb5e1fa-6378-4470-9c59-d37629eb99f4-signing-key\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:31.913315 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.913196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cbb5e1fa-6378-4470-9c59-d37629eb99f4-signing-cabundle\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:31.913315 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:31.913272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwgd\" (UniqueName: \"kubernetes.io/projected/cbb5e1fa-6378-4470-9c59-d37629eb99f4-kube-api-access-nwwgd\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.014160 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.014111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cbb5e1fa-6378-4470-9c59-d37629eb99f4-signing-cabundle\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.014307 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.014178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwgd\" (UniqueName: \"kubernetes.io/projected/cbb5e1fa-6378-4470-9c59-d37629eb99f4-kube-api-access-nwwgd\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.014307 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.014223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cbb5e1fa-6378-4470-9c59-d37629eb99f4-signing-key\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.014733 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.014708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cbb5e1fa-6378-4470-9c59-d37629eb99f4-signing-cabundle\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.016828 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.016791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cbb5e1fa-6378-4470-9c59-d37629eb99f4-signing-key\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.024846 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.024820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwgd\" (UniqueName: \"kubernetes.io/projected/cbb5e1fa-6378-4470-9c59-d37629eb99f4-kube-api-access-nwwgd\") pod \"service-ca-568869fdc5-b6vfj\" (UID: \"cbb5e1fa-6378-4470-9c59-d37629eb99f4\") " pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.146659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.146567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-b6vfj"
Sep 17 15:40:32.419289 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:32.419248    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-b6vfj"]
Sep 17 15:40:32.421618 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:32.421595    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb5e1fa_6378_4470_9c59_d37629eb99f4.slice/crio-48e98ea61faa5d9197fef22d9ffa29da9b3d345aa910cf1ae9aaa51a08ff4335 WatchSource:0}: Error finding container 48e98ea61faa5d9197fef22d9ffa29da9b3d345aa910cf1ae9aaa51a08ff4335: Status 404 returned error can't find the container with id 48e98ea61faa5d9197fef22d9ffa29da9b3d345aa910cf1ae9aaa51a08ff4335
Sep 17 15:40:33.372629 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:33.372583    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-b6vfj" event={"ID":"cbb5e1fa-6378-4470-9c59-d37629eb99f4","Type":"ContainerStarted","Data":"48e98ea61faa5d9197fef22d9ffa29da9b3d345aa910cf1ae9aaa51a08ff4335"}
Sep 17 15:40:34.377431 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:34.377401    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-b6vfj" event={"ID":"cbb5e1fa-6378-4470-9c59-d37629eb99f4","Type":"ContainerStarted","Data":"108a5a7f9cac42cb1eafc89037a4400184e4bcdcd9d4b0331186c639949e1896"}
Sep 17 15:40:34.394926 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:34.394880    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-b6vfj" podStartSLOduration=1.6297622980000002 podStartE2EDuration="3.394864865s" totalImagesPullingTime="1.765102567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:34.394312837 +0000 UTC m=+81.824476617" watchObservedRunningTime="2026-09-17 15:40:34.394864865 +0000 UTC m=+81.825028646"
Sep 17 15:40:35.440659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:35.440355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:35.441108 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:35.440731    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:35.441108 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:40:35.440790    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls podName:4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:51.440772322 +0000 UTC m=+98.870936083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mrj2w" (UID: "4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:35.441242 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:35.441137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:35.442033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:35.442012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:36.750703 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:36.750668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:36.753196 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:36.753179    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ffecac2-0ff5-4c1d-b6a3-c6c26640664b-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-c5kf6\" (UID: \"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:36.928303 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:36.928276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"
Sep 17 15:40:37.199108 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:37.199055    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6"]
Sep 17 15:40:37.388852 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:37.388814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6" event={"ID":"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b","Type":"ContainerStarted","Data":"ba9ae0af43a929e43703f557b3b24c7516c5878001c8940f64a25a5ad563224e"}
Sep 17 15:40:39.396688 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:39.396651    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6" event={"ID":"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b","Type":"ContainerStarted","Data":"e6cee868197dc0b59b5bbd47c1601fa0e0c5d883cea56fe8e7c8f6cdea7d01c3"}
Sep 17 15:40:39.397088 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:39.396705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6" event={"ID":"9ffecac2-0ff5-4c1d-b6a3-c6c26640664b","Type":"ContainerStarted","Data":"4791084f2ba4612fff62a08556ac01adbc80c6104080cf7078bf2de70878933a"}
Sep 17 15:40:39.418361 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:39.418317    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-c5kf6" podStartSLOduration=17.878683351 podStartE2EDuration="19.418303631s" totalImagesPullingTime="1.53962028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:39.417469769 +0000 UTC m=+86.847633550" watchObservedRunningTime="2026-09-17 15:40:39.418303631 +0000 UTC m=+86.848467460"
Sep 17 15:40:49.941610 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.941575    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"]
Sep 17 15:40:49.945746 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.945730    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:49.949556 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.949517    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-25nqc\""
Sep 17 15:40:49.955355 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.955333    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:40:49.956074 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.956057    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:40:49.956197 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.956061    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:40:49.956197 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.956123    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 15:40:49.956386 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.956367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"]
Sep 17 15:40:49.965069 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.965044    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"]
Sep 17 15:40:49.970843 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.970826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:49.974620 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.974598    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 15:40:49.974742 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.974632    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 15:40:49.974742 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.974647    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 15:40:49.974742 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.974609    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 15:40:49.985122 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:49.985101    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"]
Sep 17 15:40:50.055224 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-ca\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.055327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.055327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2c6n\" (UniqueName: \"kubernetes.io/projected/3858d873-ea90-4766-83e1-cea93d12acdb-kube-api-access-n2c6n\") pod \"managed-serviceaccount-addon-agent-7f587d56c6-55ngh\" (UID: \"3858d873-ea90-4766-83e1-cea93d12acdb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.055327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcprj\" (UniqueName: \"kubernetes.io/projected/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-kube-api-access-dcprj\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.055430 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055390    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-hub\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.055430 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.055490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3858d873-ea90-4766-83e1-cea93d12acdb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f587d56c6-55ngh\" (UID: \"3858d873-ea90-4766-83e1-cea93d12acdb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.055490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.055462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.155727 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.155696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-hub\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.155727 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.155731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.155943 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.155747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3858d873-ea90-4766-83e1-cea93d12acdb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f587d56c6-55ngh\" (UID: \"3858d873-ea90-4766-83e1-cea93d12acdb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.155943 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.155763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.155943 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.155934    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-ca\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.156190 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.156001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.156190 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.156032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2c6n\" (UniqueName: \"kubernetes.io/projected/3858d873-ea90-4766-83e1-cea93d12acdb-kube-api-access-n2c6n\") pod \"managed-serviceaccount-addon-agent-7f587d56c6-55ngh\" (UID: \"3858d873-ea90-4766-83e1-cea93d12acdb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.156190 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.156108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcprj\" (UniqueName: \"kubernetes.io/projected/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-kube-api-access-dcprj\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.156507 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.156479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.158500 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.158480    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-hub\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.159202 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.159173    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-ca\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.159306 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.159291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.159367 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.159343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.159447 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.159432    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3858d873-ea90-4766-83e1-cea93d12acdb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f587d56c6-55ngh\" (UID: \"3858d873-ea90-4766-83e1-cea93d12acdb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.170359 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.170332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcprj\" (UniqueName: \"kubernetes.io/projected/da6e0bd9-e7a3-40a7-8fb5-539d69ade38a-kube-api-access-dcprj\") pod \"cluster-proxy-proxy-agent-7db5869b8d-jsvxm\" (UID: \"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.170437 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.170386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2c6n\" (UniqueName: \"kubernetes.io/projected/3858d873-ea90-4766-83e1-cea93d12acdb-kube-api-access-n2c6n\") pod \"managed-serviceaccount-addon-agent-7f587d56c6-55ngh\" (UID: \"3858d873-ea90-4766-83e1-cea93d12acdb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.261428 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.261400    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"
Sep 17 15:40:50.280196 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.280172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"
Sep 17 15:40:50.559912 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.559838    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh"]
Sep 17 15:40:50.563293 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:50.563267    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3858d873_ea90_4766_83e1_cea93d12acdb.slice/crio-911eec72e1834ad2f70df84b032f4c3227cbcfe9626e88b05dd455833f589a87 WatchSource:0}: Error finding container 911eec72e1834ad2f70df84b032f4c3227cbcfe9626e88b05dd455833f589a87: Status 404 returned error can't find the container with id 911eec72e1834ad2f70df84b032f4c3227cbcfe9626e88b05dd455833f589a87
Sep 17 15:40:50.602211 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.602039    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm"]
Sep 17 15:40:50.659694 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.659658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:40:50.662040 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.662021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/752d9f57-ac3c-4115-9b3a-ceb4c4fb58de-metrics-tls\") pod \"dns-default-5rnnw\" (UID: \"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de\") " pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:40:50.690571 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.690544    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxcdv\""
Sep 17 15:40:50.698461 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.698443    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:40:50.968513 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:50.968481    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5rnnw"]
Sep 17 15:40:50.970871 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:50.970849    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752d9f57_ac3c_4115_9b3a_ceb4c4fb58de.slice/crio-1323e2cb1a64a7df442aab48bc38fd0adcd4482aa46c001680bcb073c1f179ed WatchSource:0}: Error finding container 1323e2cb1a64a7df442aab48bc38fd0adcd4482aa46c001680bcb073c1f179ed: Status 404 returned error can't find the container with id 1323e2cb1a64a7df442aab48bc38fd0adcd4482aa46c001680bcb073c1f179ed
Sep 17 15:40:51.440434 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:51.440376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rnnw" event={"ID":"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de","Type":"ContainerStarted","Data":"1323e2cb1a64a7df442aab48bc38fd0adcd4482aa46c001680bcb073c1f179ed"}
Sep 17 15:40:51.442143 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:51.442110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm" event={"ID":"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a","Type":"ContainerStarted","Data":"464b0bf114facd335c18fdadc6b8cdfd22845039548f558632439b3777131617"}
Sep 17 15:40:51.444462 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:51.444433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh" event={"ID":"3858d873-ea90-4766-83e1-cea93d12acdb","Type":"ContainerStarted","Data":"911eec72e1834ad2f70df84b032f4c3227cbcfe9626e88b05dd455833f589a87"}
Sep 17 15:40:51.467032 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:51.466995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:51.485694 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:51.485656    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mrj2w\" (UID: \"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4\") " pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:51.772035 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:51.771922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mrj2w"
Sep 17 15:40:52.082725 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:52.082587    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mrj2w"]
Sep 17 15:40:52.086576 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:52.086501    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bba0f0a_9d7e_4ee8_9f1a_1bb1d0d018c4.slice/crio-372ab1d77e68c94b9a9c580d9c18322d5d068034590b69d89afcee3d4d574608 WatchSource:0}: Error finding container 372ab1d77e68c94b9a9c580d9c18322d5d068034590b69d89afcee3d4d574608: Status 404 returned error can't find the container with id 372ab1d77e68c94b9a9c580d9c18322d5d068034590b69d89afcee3d4d574608
Sep 17 15:40:52.451649 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:52.451559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrj2w" event={"ID":"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4","Type":"ContainerStarted","Data":"bc17e2aafcbbb14b20b16dd845af4f7c8e79131d6ac599b73d8eb46c59c72179"}
Sep 17 15:40:52.451799 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:52.451644    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrj2w" event={"ID":"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4","Type":"ContainerStarted","Data":"372ab1d77e68c94b9a9c580d9c18322d5d068034590b69d89afcee3d4d574608"}
Sep 17 15:40:55.343885 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.343856    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cpn9v"
Sep 17 15:40:55.464615 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.464573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrj2w" event={"ID":"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4","Type":"ContainerStarted","Data":"2832b9398b5918eaf61e463745041fbbf9b26bb8348f5e769bc01db8665fd770"}
Sep 17 15:40:55.466403 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.466378    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rnnw" event={"ID":"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de","Type":"ContainerStarted","Data":"59c39af6ee155533db88f8d33a0def214ac74c11a539bc6ec2b3398ad1f0be44"}
Sep 17 15:40:55.466508 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.466425    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5rnnw" event={"ID":"752d9f57-ac3c-4115-9b3a-ceb4c4fb58de","Type":"ContainerStarted","Data":"3b7e1f00d93efb9c75c693f8bbaf128ccfba480ba2e7c6dcdac8e1a6b104e650"}
Sep 17 15:40:55.466508 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.466479    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:40:55.468536 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.468515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm" event={"ID":"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a","Type":"ContainerStarted","Data":"4921f10973de194b140345a2e10424123f09b0bb5f2d5eaab52f78ebf5fe1127"}
Sep 17 15:40:55.468633 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.468545    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm" event={"ID":"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a","Type":"ContainerStarted","Data":"d4719d51cf0aca02a3879f0427e6a62a017a02fb17fb17be4fd4161007f9a0a4"}
Sep 17 15:40:55.468633 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.468558    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm" event={"ID":"da6e0bd9-e7a3-40a7-8fb5-539d69ade38a","Type":"ContainerStarted","Data":"8cadaf75aab69990583c520761da4b60cbfd47936ab68afa4118aa3e480ba3ea"}
Sep 17 15:40:55.469999 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.469977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh" event={"ID":"3858d873-ea90-4766-83e1-cea93d12acdb","Type":"ContainerStarted","Data":"3f70ef1e6d92a615c60ccaad27886e36a6fb60b50759423659d7cad1d62c2580"}
Sep 17 15:40:55.484786 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.484737    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5rnnw" podStartSLOduration=65.374660798 podStartE2EDuration="1m9.484721615s" totalImagesPullingTime="4.110060817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:55.484449541 +0000 UTC m=+102.914613324" watchObservedRunningTime="2026-09-17 15:40:55.484721615 +0000 UTC m=+102.914885397"
Sep 17 15:40:55.504684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.504614    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f587d56c6-55ngh" podStartSLOduration=1.988552478 podStartE2EDuration="6.504600482s" totalImagesPullingTime="4.516048004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:55.50390294 +0000 UTC m=+102.934066721" watchObservedRunningTime="2026-09-17 15:40:55.504600482 +0000 UTC m=+102.934764263"
Sep 17 15:40:55.529667 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:55.529621    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7db5869b8d-jsvxm" podStartSLOduration=2.059144345 podStartE2EDuration="6.529606528s" totalImagesPullingTime="4.470462183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:55.52938025 +0000 UTC m=+102.959544030" watchObservedRunningTime="2026-09-17 15:40:55.529606528 +0000 UTC m=+102.959770310"
Sep 17 15:40:57.479536 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.479498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mrj2w" event={"ID":"4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4","Type":"ContainerStarted","Data":"ddc1afc31f25f3a0cd3e05338c41c556542ceff2992e62e385a69d1480f01963"}
Sep 17 15:40:57.503429 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.503378    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mrj2w" podStartSLOduration=33.845400264 podStartE2EDuration="38.503362832s" totalImagesPullingTime="4.657962568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:57.502899123 +0000 UTC m=+104.933062905" watchObservedRunningTime="2026-09-17 15:40:57.503362832 +0000 UTC m=+104.933526612"
Sep 17 15:40:57.901366 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.901336    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rtcld"]
Sep 17 15:40:57.916815 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.916789    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:57.919834 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.919647    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:40:57.919834 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.919683    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:40:57.919834 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.919686    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:40:57.919834 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.919725    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:40:57.919834 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.919654    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:40:57.920662 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.920641    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8hqsc\""
Sep 17 15:40:57.920772 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:57.920662    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:40:58.024399 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef6c41bf-5cd5-4229-bd2e-32d531171207-metrics-client-ca\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024590 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-root\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024590 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024464    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-sys\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024701 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-textfile\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024701 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024630    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-accelerators-collector-config\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024701 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8t2p\" (UniqueName: \"kubernetes.io/projected/ef6c41bf-5cd5-4229-bd2e-32d531171207-kube-api-access-z8t2p\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024701 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-tls\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024887 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-wtmp\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.024887 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.024775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125359 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125319    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-wtmp\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125519 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125390    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125519 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef6c41bf-5cd5-4229-bd2e-32d531171207-metrics-client-ca\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125519 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-root\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-sys\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125528    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-wtmp\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-textfile\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-sys\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-accelerators-collector-config\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8t2p\" (UniqueName: \"kubernetes.io/projected/ef6c41bf-5cd5-4229-bd2e-32d531171207-kube-api-access-z8t2p\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.125978 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.125693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-tls\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.126563 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.126056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef6c41bf-5cd5-4229-bd2e-32d531171207-root\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.126563 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.126215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef6c41bf-5cd5-4229-bd2e-32d531171207-metrics-client-ca\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.126563 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.126496    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-textfile\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.126778 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.126582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-accelerators-collector-config\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.128820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.128798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.128976 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.128938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef6c41bf-5cd5-4229-bd2e-32d531171207-node-exporter-tls\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.135693 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.135670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8t2p\" (UniqueName: \"kubernetes.io/projected/ef6c41bf-5cd5-4229-bd2e-32d531171207-kube-api-access-z8t2p\") pod \"node-exporter-rtcld\" (UID: \"ef6c41bf-5cd5-4229-bd2e-32d531171207\") " pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.227962 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.227858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rtcld"
Sep 17 15:40:58.257522 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:40:58.257491    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6c41bf_5cd5_4229_bd2e_32d531171207.slice/crio-e1e070bb516872d0096daf276e6e1dffba6235beedd80de6dbc0fd139f2b6c8c WatchSource:0}: Error finding container e1e070bb516872d0096daf276e6e1dffba6235beedd80de6dbc0fd139f2b6c8c: Status 404 returned error can't find the container with id e1e070bb516872d0096daf276e6e1dffba6235beedd80de6dbc0fd139f2b6c8c
Sep 17 15:40:58.484285 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:58.484198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtcld" event={"ID":"ef6c41bf-5cd5-4229-bd2e-32d531171207","Type":"ContainerStarted","Data":"e1e070bb516872d0096daf276e6e1dffba6235beedd80de6dbc0fd139f2b6c8c"}
Sep 17 15:40:59.489491 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:40:59.489459    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtcld" event={"ID":"ef6c41bf-5cd5-4229-bd2e-32d531171207","Type":"ContainerStarted","Data":"36ad20b3309d25a620b8c4a63f13e88658c5c881ea523babe40a06216ff9d80e"}
Sep 17 15:41:00.494702 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:00.494664    2575 generic.go:356] "Generic (PLEG): container finished" podID="ef6c41bf-5cd5-4229-bd2e-32d531171207" containerID="36ad20b3309d25a620b8c4a63f13e88658c5c881ea523babe40a06216ff9d80e" exitCode=0
Sep 17 15:41:00.495082 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:00.494729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtcld" event={"ID":"ef6c41bf-5cd5-4229-bd2e-32d531171207","Type":"ContainerDied","Data":"36ad20b3309d25a620b8c4a63f13e88658c5c881ea523babe40a06216ff9d80e"}
Sep 17 15:41:01.500678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:01.500639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtcld" event={"ID":"ef6c41bf-5cd5-4229-bd2e-32d531171207","Type":"ContainerStarted","Data":"b172ee2dc0a8be99e886e10b0e5555e3a9d018039d7be03bf96c683a51d6650a"}
Sep 17 15:41:01.501094 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:01.500703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rtcld" event={"ID":"ef6c41bf-5cd5-4229-bd2e-32d531171207","Type":"ContainerStarted","Data":"fefbeded70676167748af5f1358230f9184e658d6a526911dd24e23fa13f81b1"}
Sep 17 15:41:01.523863 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:01.523810    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rtcld" podStartSLOduration=3.356324251 podStartE2EDuration="4.523796294s" totalImagesPullingTime="1.07305065s" totalInitContainerRuntime="94.421393ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:01.522158546 +0000 UTC m=+108.952322328" watchObservedRunningTime="2026-09-17 15:41:01.523796294 +0000 UTC m=+108.953960075"
Sep 17 15:41:05.476259 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:05.476231    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5rnnw"
Sep 17 15:41:22.908477 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:22.908433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:41:22.910816 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:22.910794    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa01c61-e6a5-47d6-8a03-da7e34a2cee4-metrics-certs\") pod \"network-metrics-daemon-wws4p\" (UID: \"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4\") " pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:41:23.069025 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:23.068998    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm2sj\""
Sep 17 15:41:23.077166 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:23.077150    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wws4p"
Sep 17 15:41:23.346118 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:23.346007    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wws4p"]
Sep 17 15:41:23.348467 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:41:23.348442    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa01c61_e6a5_47d6_8a03_da7e34a2cee4.slice/crio-1eb2fe32d91ace3f40f46cac906cf2cc38938c83e401e06fcd7ff9f8d1948712 WatchSource:0}: Error finding container 1eb2fe32d91ace3f40f46cac906cf2cc38938c83e401e06fcd7ff9f8d1948712: Status 404 returned error can't find the container with id 1eb2fe32d91ace3f40f46cac906cf2cc38938c83e401e06fcd7ff9f8d1948712
Sep 17 15:41:23.574903 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:23.574872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wws4p" event={"ID":"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4","Type":"ContainerStarted","Data":"1eb2fe32d91ace3f40f46cac906cf2cc38938c83e401e06fcd7ff9f8d1948712"}
Sep 17 15:41:24.580298 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:24.580262    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wws4p" event={"ID":"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4","Type":"ContainerStarted","Data":"f08084e9a155392665bd36a2fa48e260f56f89cd32a2a0aa4c97930e178508f1"}
Sep 17 15:41:24.580710 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:24.580323    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wws4p" event={"ID":"2aa01c61-e6a5-47d6-8a03-da7e34a2cee4","Type":"ContainerStarted","Data":"769e4428a53edaeccd05e0245b8fc3bcdf8691d66e6299e580f0ab62f079f07c"}
Sep 17 15:41:34.616619 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:34.616582    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ed0dd50-7be8-49e8-a57e-14e04452e87f" containerID="e04680a7e17aa7f708ca1891910ba7aab87a40efcc0ac5dd2e4a88da42dca8a6" exitCode=0
Sep 17 15:41:34.617033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:34.616623    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gvzhp" event={"ID":"5ed0dd50-7be8-49e8-a57e-14e04452e87f","Type":"ContainerDied","Data":"e04680a7e17aa7f708ca1891910ba7aab87a40efcc0ac5dd2e4a88da42dca8a6"}
Sep 17 15:41:34.617033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:34.617001    2575 scope.go:117] "RemoveContainer" containerID="e04680a7e17aa7f708ca1891910ba7aab87a40efcc0ac5dd2e4a88da42dca8a6"
Sep 17 15:41:34.670284 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:34.670243    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wws4p" podStartSLOduration=140.882139624 podStartE2EDuration="2m21.67021365s" totalImagesPullingTime="788.074026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:24.599556991 +0000 UTC m=+132.029720772" watchObservedRunningTime="2026-09-17 15:41:34.67021365 +0000 UTC m=+142.100377427"
Sep 17 15:41:35.623100 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:35.623060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gvzhp" event={"ID":"5ed0dd50-7be8-49e8-a57e-14e04452e87f","Type":"ContainerStarted","Data":"9235001d6091b40dbec9a4968b50a69ce6439f193bbb2f1adcac8e031e06f3ad"}
Sep 17 15:41:44.656490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:44.656457    2575 generic.go:356] "Generic (PLEG): container finished" podID="7bf6160d-e4bf-4e3c-9884-59ecab0f839d" containerID="d690a78291eedab2704b4ad70b9aa6bffdf6304cd8bffae7acfb0c3221b8317f" exitCode=0
Sep 17 15:41:44.656926 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:44.656515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8" event={"ID":"7bf6160d-e4bf-4e3c-9884-59ecab0f839d","Type":"ContainerDied","Data":"d690a78291eedab2704b4ad70b9aa6bffdf6304cd8bffae7acfb0c3221b8317f"}
Sep 17 15:41:44.657042 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:44.656927    2575 scope.go:117] "RemoveContainer" containerID="d690a78291eedab2704b4ad70b9aa6bffdf6304cd8bffae7acfb0c3221b8317f"
Sep 17 15:41:45.661120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:41:45.661089    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n44s8" event={"ID":"7bf6160d-e4bf-4e3c-9884-59ecab0f839d","Type":"ContainerStarted","Data":"ef87e3d3430267ea7531db89123bb6db7493e34aa9fb6dc20693bb13d31eec13"}
Sep 17 15:44:12.942821 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:44:12.942777    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:45:30.828690 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.828655    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-55hzl"]
Sep 17 15:45:30.831771 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.831751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:30.836210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.836188    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 15:45:30.836328 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.836236    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 15:45:30.836328 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.836245    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 15:45:30.837094 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.837071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 15:45:30.837218 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.837071    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nc6c6\""
Sep 17 15:45:30.837279 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.837071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 15:45:30.844839 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:30.844818    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-55hzl"]
Sep 17 15:45:31.008706 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.008678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-cabundle0\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.008838 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.008734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-certificates\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.008838 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.008752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmv7\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-kube-api-access-rzmv7\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.109598 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.109536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-certificates\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.109598 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.109575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmv7\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-kube-api-access-rzmv7\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.109743 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.109600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-cabundle0\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.109743 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:45:31.109719    2575 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 15:45:31.109743 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:45:31.109743    2575 secret.go:281] references non-existent secret key: ca.crt
Sep 17 15:45:31.109832 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:45:31.109755    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 15:45:31.109832 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:45:31.109771    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-55hzl: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 15:45:31.109893 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:45:31.109839    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-certificates podName:15e09b78-3b10-4cb9-9043-24fd9c9b4d9c nodeName:}" failed. No retries permitted until 2026-09-17 15:45:31.609820175 +0000 UTC m=+379.039983948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-certificates") pod "keda-operator-579c8958f7-55hzl" (UID: "15e09b78-3b10-4cb9-9043-24fd9c9b4d9c") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 15:45:31.110131 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.110115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-cabundle0\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.120401 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.120375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmv7\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-kube-api-access-rzmv7\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.613449 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.613419    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-certificates\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.615961 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.615928    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/15e09b78-3b10-4cb9-9043-24fd9c9b4d9c-certificates\") pod \"keda-operator-579c8958f7-55hzl\" (UID: \"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c\") " pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:31.741433 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:31.741403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:32.013319 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:32.013281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-55hzl"]
Sep 17 15:45:32.016237 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:45:32.016214    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e09b78_3b10_4cb9_9043_24fd9c9b4d9c.slice/crio-9228d7c3a930b5df093e49b523c0ff8f0e5e76a7e6a3745ace61dd7c62439bd0 WatchSource:0}: Error finding container 9228d7c3a930b5df093e49b523c0ff8f0e5e76a7e6a3745ace61dd7c62439bd0: Status 404 returned error can't find the container with id 9228d7c3a930b5df093e49b523c0ff8f0e5e76a7e6a3745ace61dd7c62439bd0
Sep 17 15:45:32.017597 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:32.017576    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:45:32.437027 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:32.436934    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-55hzl" event={"ID":"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c","Type":"ContainerStarted","Data":"9228d7c3a930b5df093e49b523c0ff8f0e5e76a7e6a3745ace61dd7c62439bd0"}
Sep 17 15:45:36.453731 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:36.453698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-55hzl" event={"ID":"15e09b78-3b10-4cb9-9043-24fd9c9b4d9c","Type":"ContainerStarted","Data":"680ff4c65e281cbac6e39e69ada6288b25b776eb897c2c284c4ea705b579f9c9"}
Sep 17 15:45:36.454108 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:36.453821    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:45:36.475279 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:36.475229    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-55hzl" podStartSLOduration=2.92273552 podStartE2EDuration="6.475215668s" totalImagesPullingTime="3.552480148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:36.473909531 +0000 UTC m=+383.904073312" watchObservedRunningTime="2026-09-17 15:45:36.475215668 +0000 UTC m=+383.905379554"
Sep 17 15:45:57.459718 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:45:57.459687    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-55hzl"
Sep 17 15:46:52.725749 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.725666    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-n2l8b"]
Sep 17 15:46:52.728758 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.728737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:52.731214 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.731189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:46:52.731214 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.731191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6vf9l\""
Sep 17 15:46:52.732217 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.732200    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 15:46:52.732273 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.732261    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:46:52.737255 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.737237    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-n2l8b"]
Sep 17 15:46:52.874673 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.874648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/98550bc3-6fd4-403d-96a6-189217ab7a8c-data\") pod \"seaweedfs-64568bcd49-n2l8b\" (UID: \"98550bc3-6fd4-403d-96a6-189217ab7a8c\") " pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:52.874792 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.874698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgrc\" (UniqueName: \"kubernetes.io/projected/98550bc3-6fd4-403d-96a6-189217ab7a8c-kube-api-access-9wgrc\") pod \"seaweedfs-64568bcd49-n2l8b\" (UID: \"98550bc3-6fd4-403d-96a6-189217ab7a8c\") " pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:52.975533 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.975507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgrc\" (UniqueName: \"kubernetes.io/projected/98550bc3-6fd4-403d-96a6-189217ab7a8c-kube-api-access-9wgrc\") pod \"seaweedfs-64568bcd49-n2l8b\" (UID: \"98550bc3-6fd4-403d-96a6-189217ab7a8c\") " pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:52.975662 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.975545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/98550bc3-6fd4-403d-96a6-189217ab7a8c-data\") pod \"seaweedfs-64568bcd49-n2l8b\" (UID: \"98550bc3-6fd4-403d-96a6-189217ab7a8c\") " pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:52.975900 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.975856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/98550bc3-6fd4-403d-96a6-189217ab7a8c-data\") pod \"seaweedfs-64568bcd49-n2l8b\" (UID: \"98550bc3-6fd4-403d-96a6-189217ab7a8c\") " pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:52.985627 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:52.985606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgrc\" (UniqueName: \"kubernetes.io/projected/98550bc3-6fd4-403d-96a6-189217ab7a8c-kube-api-access-9wgrc\") pod \"seaweedfs-64568bcd49-n2l8b\" (UID: \"98550bc3-6fd4-403d-96a6-189217ab7a8c\") " pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:53.038556 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:53.038534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:53.308543 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:53.308512    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-n2l8b"]
Sep 17 15:46:53.311681 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:46:53.311654    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98550bc3_6fd4_403d_96a6_189217ab7a8c.slice/crio-b97585027654d55548179f5380b840f180b4c6884f14536d6aae5d3632a4e65b WatchSource:0}: Error finding container b97585027654d55548179f5380b840f180b4c6884f14536d6aae5d3632a4e65b: Status 404 returned error can't find the container with id b97585027654d55548179f5380b840f180b4c6884f14536d6aae5d3632a4e65b
Sep 17 15:46:53.714238 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:53.714150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-n2l8b" event={"ID":"98550bc3-6fd4-403d-96a6-189217ab7a8c","Type":"ContainerStarted","Data":"b97585027654d55548179f5380b840f180b4c6884f14536d6aae5d3632a4e65b"}
Sep 17 15:46:57.729620 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:57.729582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-n2l8b" event={"ID":"98550bc3-6fd4-403d-96a6-189217ab7a8c","Type":"ContainerStarted","Data":"8e0964991d1584c44c1106d49f9a37cd4607fbac3b1ac4321d5e3d8fc056b9ab"}
Sep 17 15:46:57.730047 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:57.729675    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:46:57.748678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:46:57.748638    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-n2l8b" podStartSLOduration=2.264376682 podStartE2EDuration="5.748624702s" totalImagesPullingTime="3.48424802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:46:57.748373365 +0000 UTC m=+465.178537147" watchObservedRunningTime="2026-09-17 15:46:57.748624702 +0000 UTC m=+465.178788482"
Sep 17 15:47:03.735307 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:47:03.735275    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-n2l8b"
Sep 17 15:48:12.026438 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.026254    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-n2jds"]
Sep 17 15:48:12.029187 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.029166    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.032216 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.032193    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 15:48:12.032724 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.032697    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6l8wp\""
Sep 17 15:48:12.033083 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.033061    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n2jds"]
Sep 17 15:48:12.036177 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.036158    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-vjl29"]
Sep 17 15:48:12.038780 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.038726    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.041077 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.040880    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 15:48:12.041077 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.040919    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-hscmq\""
Sep 17 15:48:12.041077 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.040935    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 15:48:12.048912 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.048889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vjl29"]
Sep 17 15:48:12.114015 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.113991    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-metrics-certs\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.114128 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.114041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8kd\" (UniqueName: \"kubernetes.io/projected/1e3951de-3730-47ef-bf7a-49b1d22f3571-kube-api-access-pl8kd\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.114128 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.114072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8tz\" (UniqueName: \"kubernetes.io/projected/4cc6ff97-b737-470f-8e1c-a9b1d82c992c-kube-api-access-jq8tz\") pod \"model-serving-api-7b954ffc95-n2jds\" (UID: \"4cc6ff97-b737-470f-8e1c-a9b1d82c992c\") " pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.114128 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.114087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-cert\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.114128 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.114111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6ff97-b737-470f-8e1c-a9b1d82c992c-tls-certs\") pod \"model-serving-api-7b954ffc95-n2jds\" (UID: \"4cc6ff97-b737-470f-8e1c-a9b1d82c992c\") " pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.215220 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.215195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-metrics-certs\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.215332 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.215250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8kd\" (UniqueName: \"kubernetes.io/projected/1e3951de-3730-47ef-bf7a-49b1d22f3571-kube-api-access-pl8kd\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.215332 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.215283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8tz\" (UniqueName: \"kubernetes.io/projected/4cc6ff97-b737-470f-8e1c-a9b1d82c992c-kube-api-access-jq8tz\") pod \"model-serving-api-7b954ffc95-n2jds\" (UID: \"4cc6ff97-b737-470f-8e1c-a9b1d82c992c\") " pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.215332 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.215307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-cert\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.215482 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.215335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6ff97-b737-470f-8e1c-a9b1d82c992c-tls-certs\") pod \"model-serving-api-7b954ffc95-n2jds\" (UID: \"4cc6ff97-b737-470f-8e1c-a9b1d82c992c\") " pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.215482 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:48:12.215343    2575 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 15:48:12.215482 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:48:12.215402    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-metrics-certs podName:1e3951de-3730-47ef-bf7a-49b1d22f3571 nodeName:}" failed. No retries permitted until 2026-09-17 15:48:12.715381968 +0000 UTC m=+540.145545733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-metrics-certs") pod "odh-model-controller-7b966d7984-vjl29" (UID: "1e3951de-3730-47ef-bf7a-49b1d22f3571") : secret "odh-model-controller-metrics-tls" not found
Sep 17 15:48:12.215482 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:48:12.215472    2575 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 15:48:12.215691 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:48:12.215550    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-cert podName:1e3951de-3730-47ef-bf7a-49b1d22f3571 nodeName:}" failed. No retries permitted until 2026-09-17 15:48:12.715532896 +0000 UTC m=+540.145696660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-cert") pod "odh-model-controller-7b966d7984-vjl29" (UID: "1e3951de-3730-47ef-bf7a-49b1d22f3571") : secret "odh-model-controller-webhook-cert" not found
Sep 17 15:48:12.218001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.217977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc6ff97-b737-470f-8e1c-a9b1d82c992c-tls-certs\") pod \"model-serving-api-7b954ffc95-n2jds\" (UID: \"4cc6ff97-b737-470f-8e1c-a9b1d82c992c\") " pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.224981 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.224933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8kd\" (UniqueName: \"kubernetes.io/projected/1e3951de-3730-47ef-bf7a-49b1d22f3571-kube-api-access-pl8kd\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.225659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.225638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8tz\" (UniqueName: \"kubernetes.io/projected/4cc6ff97-b737-470f-8e1c-a9b1d82c992c-kube-api-access-jq8tz\") pod \"model-serving-api-7b954ffc95-n2jds\" (UID: \"4cc6ff97-b737-470f-8e1c-a9b1d82c992c\") " pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.343198 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.343113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:12.615938 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.615853    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n2jds"]
Sep 17 15:48:12.619483 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:48:12.619460    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc6ff97_b737_470f_8e1c_a9b1d82c992c.slice/crio-e15df8b10887eff1875fed7e5a0a89c6fb26f8cd2703c16f3cc466cce0bb0c86 WatchSource:0}: Error finding container e15df8b10887eff1875fed7e5a0a89c6fb26f8cd2703c16f3cc466cce0bb0c86: Status 404 returned error can't find the container with id e15df8b10887eff1875fed7e5a0a89c6fb26f8cd2703c16f3cc466cce0bb0c86
Sep 17 15:48:12.718930 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.718895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-cert\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.719074 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.718977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-metrics-certs\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.721385 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.721362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-cert\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.721474 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.721456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e3951de-3730-47ef-bf7a-49b1d22f3571-metrics-certs\") pod \"odh-model-controller-7b966d7984-vjl29\" (UID: \"1e3951de-3730-47ef-bf7a-49b1d22f3571\") " pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.959410 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.959337    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-hscmq\""
Sep 17 15:48:12.968094 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.968069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:12.993738 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:12.993709    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n2jds" event={"ID":"4cc6ff97-b737-470f-8e1c-a9b1d82c992c","Type":"ContainerStarted","Data":"e15df8b10887eff1875fed7e5a0a89c6fb26f8cd2703c16f3cc466cce0bb0c86"}
Sep 17 15:48:13.245302 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:13.245278    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vjl29"]
Sep 17 15:48:13.247606 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:48:13.247566    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3951de_3730_47ef_bf7a_49b1d22f3571.slice/crio-8003376fb465892d018ba4509a48f2ac6d7b3d3c301f011259b868029e7e1880 WatchSource:0}: Error finding container 8003376fb465892d018ba4509a48f2ac6d7b3d3c301f011259b868029e7e1880: Status 404 returned error can't find the container with id 8003376fb465892d018ba4509a48f2ac6d7b3d3c301f011259b868029e7e1880
Sep 17 15:48:14.000756 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:14.000709    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vjl29" event={"ID":"1e3951de-3730-47ef-bf7a-49b1d22f3571","Type":"ContainerStarted","Data":"8003376fb465892d018ba4509a48f2ac6d7b3d3c301f011259b868029e7e1880"}
Sep 17 15:48:17.013727 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:17.013689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n2jds" event={"ID":"4cc6ff97-b737-470f-8e1c-a9b1d82c992c","Type":"ContainerStarted","Data":"43d094f48adcf0e6362c0dccba89bd670325bd7faee34a58d3ea1e675ccff96f"}
Sep 17 15:48:17.014195 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:17.013795    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:48:17.015089 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:17.015067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vjl29" event={"ID":"1e3951de-3730-47ef-bf7a-49b1d22f3571","Type":"ContainerStarted","Data":"9beca8a6f66bdc7f7ef7391c4b4ec60724bcf942772b7248f41e8688a26cc6e2"}
Sep 17 15:48:17.015202 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:17.015177    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:17.032315 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:17.032275    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-n2jds" podStartSLOduration=1.195209817 podStartE2EDuration="5.032260864s" totalImagesPullingTime="3.837051047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:17.032233539 +0000 UTC m=+544.462397321" watchObservedRunningTime="2026-09-17 15:48:17.032260864 +0000 UTC m=+544.462424669"
Sep 17 15:48:17.049180 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:17.049141    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-vjl29" podStartSLOduration=1.830034188 podStartE2EDuration="5.049127231s" totalImagesPullingTime="3.219093043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:17.048435265 +0000 UTC m=+544.478599046" watchObservedRunningTime="2026-09-17 15:48:17.049127231 +0000 UTC m=+544.479291013"
Sep 17 15:48:28.022290 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:28.022254    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-vjl29"
Sep 17 15:48:28.024204 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:48:28.024174    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-n2jds"
Sep 17 15:50:02.266238 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.266195    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"]
Sep 17 15:50:02.269494 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.269474    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"]
Sep 17 15:50:02.269625 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.269597    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.272165 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.272143    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 15:50:02.272425 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.272405    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:50:02.272574 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.272437    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:50:02.272574 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.272405    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:50:02.272700 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.272689    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 15:50:02.272973 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.272938    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.273209 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.273156    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:50:02.273209 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.273173    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 15:50:02.275225 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.275205    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5b828-predictor-serving-cert\""
Sep 17 15:50:02.275624 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.275604    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-5b828-kube-rbac-proxy-sar-config\""
Sep 17 15:50:02.282161 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.282129    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"]
Sep 17 15:50:02.284900 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.284880    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"]
Sep 17 15:50:02.369335 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5h22\" (UniqueName: \"kubernetes.io/projected/5fd86159-01c7-40fd-9ae7-00099204f238-kube-api-access-g5h22\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.369335 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.369558 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fg2\" (UniqueName: \"kubernetes.io/projected/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-api-access-56fg2\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.369558 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.369558 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fd86159-01c7-40fd-9ae7-00099204f238-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.369558 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fd86159-01c7-40fd-9ae7-00099204f238-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.369558 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.369558 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.369740 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.369740 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.369580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.470486 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5h22\" (UniqueName: \"kubernetes.io/projected/5fd86159-01c7-40fd-9ae7-00099204f238-kube-api-access-g5h22\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.470486 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fg2\" (UniqueName: \"kubernetes.io/projected/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-api-access-56fg2\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fd86159-01c7-40fd-9ae7-00099204f238-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470601    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fd86159-01c7-40fd-9ae7-00099204f238-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.470736 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.471154 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:50:02.470827    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-5b828-predictor-serving-cert" not found
Sep 17 15:50:02.471154 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:50:02.470915    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls podName:e08cbb18-7bb0-4981-85a4-998b8a5026ce nodeName:}" failed. No retries permitted until 2026-09-17 15:50:02.970893607 +0000 UTC m=+650.401057365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls") pod "isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" (UID: "e08cbb18-7bb0-4981-85a4-998b8a5026ce") : secret "isvc-raw-sklearn-batcher-5b828-predictor-serving-cert" not found
Sep 17 15:50:02.471154 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.470997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.471346 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.471327    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fd86159-01c7-40fd-9ae7-00099204f238-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.471518 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.471498    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.471570 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.471501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.471628 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.471603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.472003 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.471981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.473579 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.473560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fd86159-01c7-40fd-9ae7-00099204f238-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.481290 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.481265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5h22\" (UniqueName: \"kubernetes.io/projected/5fd86159-01c7-40fd-9ae7-00099204f238-kube-api-access-g5h22\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-m9mf4\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.481748 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.481726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fg2\" (UniqueName: \"kubernetes.io/projected/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-api-access-56fg2\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.583797 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.583721    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:02.865391 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.865367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"]
Sep 17 15:50:02.868182 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:50:02.868154    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd86159_01c7_40fd_9ae7_00099204f238.slice/crio-65104b2d8a59df8063203f08414c078647e2fa890a85c70ecc8993382c958b6c WatchSource:0}: Error finding container 65104b2d8a59df8063203f08414c078647e2fa890a85c70ecc8993382c958b6c: Status 404 returned error can't find the container with id 65104b2d8a59df8063203f08414c078647e2fa890a85c70ecc8993382c958b6c
Sep 17 15:50:02.973957 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.973922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:02.976501 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:02.976478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls\") pod \"isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:03.192003 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:03.191905    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:03.400249 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:03.400208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerStarted","Data":"65104b2d8a59df8063203f08414c078647e2fa890a85c70ecc8993382c958b6c"}
Sep 17 15:50:03.474499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:03.474471    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"]
Sep 17 15:50:03.476503 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:50:03.476478    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08cbb18_7bb0_4981_85a4_998b8a5026ce.slice/crio-bf6d4404e99a397d23bd10ca9e0b743bd5e19d6ceeeb8a12bbc59932aa11aff4 WatchSource:0}: Error finding container bf6d4404e99a397d23bd10ca9e0b743bd5e19d6ceeeb8a12bbc59932aa11aff4: Status 404 returned error can't find the container with id bf6d4404e99a397d23bd10ca9e0b743bd5e19d6ceeeb8a12bbc59932aa11aff4
Sep 17 15:50:04.410705 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:04.410665    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerStarted","Data":"bf6d4404e99a397d23bd10ca9e0b743bd5e19d6ceeeb8a12bbc59932aa11aff4"}
Sep 17 15:50:08.429584 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:08.429541    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerStarted","Data":"935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f"}
Sep 17 15:50:08.431285 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:08.431255    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerStarted","Data":"db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd"}
Sep 17 15:50:11.445782 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:11.445693    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerID="db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd" exitCode=0
Sep 17 15:50:11.446229 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:11.445769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerDied","Data":"db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd"}
Sep 17 15:50:11.447588 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:11.447491    2575 generic.go:356] "Generic (PLEG): container finished" podID="5fd86159-01c7-40fd-9ae7-00099204f238" containerID="935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f" exitCode=0
Sep 17 15:50:11.447588 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:11.447550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerDied","Data":"935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f"}
Sep 17 15:50:29.558849 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:29.558812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerStarted","Data":"89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f"}
Sep 17 15:50:30.565603 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:30.565570    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerStarted","Data":"a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0"}
Sep 17 15:50:32.578005 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:32.577966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerStarted","Data":"20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13"}
Sep 17 15:50:32.580529 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:32.580504    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerStarted","Data":"8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335"}
Sep 17 15:50:32.605285 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:32.605241    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podStartSLOduration=2.2534740109999998 podStartE2EDuration="30.605228108s" totalImagesPullingTime="24.690705993s" totalInitContainerRuntime="3.661048104s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:32.604343624 +0000 UTC m=+680.034507408" watchObservedRunningTime="2026-09-17 15:50:32.605228108 +0000 UTC m=+680.035391918"
Sep 17 15:50:33.590097 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:33.590062    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:33.590534 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:33.590168    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:33.591557 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:33.591521    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:50:34.591880 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.591838    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerStarted","Data":"f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4"}
Sep 17 15:50:34.592357 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.592222    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:34.592357 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.592282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:34.592357 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.592324    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:34.592357 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.592269    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:50:34.593578 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.593545    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:50:34.594235 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.594215    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:50:34.622243 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:34.622202    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podStartSLOduration=2.318159837 podStartE2EDuration="32.622189915s" totalImagesPullingTime="26.479890818s" totalInitContainerRuntime="3.82413926s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:02 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:34.619680442 +0000 UTC m=+682.049844223" watchObservedRunningTime="2026-09-17 15:50:34.622189915 +0000 UTC m=+682.052353695"
Sep 17 15:50:35.596844 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:35.596803    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:50:35.597307 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:35.597192    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:50:39.597148 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:39.597121    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:50:39.597728 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:39.597702    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:50:40.601551 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:40.601519    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:50:40.602228 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:40.602197    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:50:40.602459 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:40.602436    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:50:49.598024 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:49.597987    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:50:50.602408 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:50.602364    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:50:50.602866 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:50.602840    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:50:59.597650 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:50:59.597606    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:51:00.602355 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:00.602319    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:51:00.602876 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:00.602852    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:09.597869 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:09.597790    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:51:10.602503 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:10.602460    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:51:10.602980 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:10.602863    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:19.598195 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:19.598156    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:51:20.602671 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:20.602629    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:51:20.603141 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:20.603042    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:29.598134 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:29.598095    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:51:30.602132 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:30.602088    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:51:30.602577 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:30.602492    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:39.598125 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:39.598085    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 17 15:51:40.602771 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:40.602733    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:51:40.603250 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:40.603158    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:47.057039 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:47.057008    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:51:49.057553 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:49.057526    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:51:49.057926 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:49.057599    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:51:57.079656 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:57.079620    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"]
Sep 17 15:51:57.080086 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:57.079880    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" containerID="cri-o://a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0" gracePeriod=30
Sep 17 15:51:57.080086 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:57.079908    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" containerID="cri-o://8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335" gracePeriod=30
Sep 17 15:51:57.080086 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:57.079894    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" containerID="cri-o://f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4" gracePeriod=30
Sep 17 15:51:57.917238 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:57.917206    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerID="8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335" exitCode=2
Sep 17 15:51:57.917409 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:57.917283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerDied","Data":"8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335"}
Sep 17 15:51:59.052878 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:59.052826    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:51:59.053334 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:51:59.053148    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:00.597597 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:00.597557    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 15:52:01.933850 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:01.933820    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerID="a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0" exitCode=0
Sep 17 15:52:01.934205 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:01.933879    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerDied","Data":"a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0"}
Sep 17 15:52:05.597478 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:05.597441    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 15:52:09.052880 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:09.052834    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:52:09.053392 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:09.053172    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:10.597144 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:10.597100    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 15:52:10.597508 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:10.597240    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:52:15.597682 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:15.597640    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 15:52:19.052907 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:19.052864    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 15:52:19.053405 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:19.053175    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:19.060608 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:19.057890    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:52:19.060608 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:19.058082    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:52:20.597940 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:20.597898    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 15:52:25.597472 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:25.597429    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 15:52:27.217804 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.217780    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:52:27.393766 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.393690    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-rbac-proxy-sar-config\") pod \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") "
Sep 17 15:52:27.393766 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.393734    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56fg2\" (UniqueName: \"kubernetes.io/projected/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-api-access-56fg2\") pod \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") "
Sep 17 15:52:27.393766 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.393762    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-cabundle-cert\") pod \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") "
Sep 17 15:52:27.394023 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.393789    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kserve-provision-location\") pod \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") "
Sep 17 15:52:27.394023 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.393819    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls\") pod \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\" (UID: \"e08cbb18-7bb0-4981-85a4-998b8a5026ce\") "
Sep 17 15:52:27.394191 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.394164    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e08cbb18-7bb0-4981-85a4-998b8a5026ce" (UID: "e08cbb18-7bb0-4981-85a4-998b8a5026ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:27.394260 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.394179    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e08cbb18-7bb0-4981-85a4-998b8a5026ce" (UID: "e08cbb18-7bb0-4981-85a4-998b8a5026ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:27.394260 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.394207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e08cbb18-7bb0-4981-85a4-998b8a5026ce" (UID: "e08cbb18-7bb0-4981-85a4-998b8a5026ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:27.396189 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.396164    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-api-access-56fg2" (OuterVolumeSpecName: "kube-api-access-56fg2") pod "e08cbb18-7bb0-4981-85a4-998b8a5026ce" (UID: "e08cbb18-7bb0-4981-85a4-998b8a5026ce"). InnerVolumeSpecName "kube-api-access-56fg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:27.396289 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.396164    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e08cbb18-7bb0-4981-85a4-998b8a5026ce" (UID: "e08cbb18-7bb0-4981-85a4-998b8a5026ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:52:27.494478 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.494445    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:52:27.494478 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.494472    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e08cbb18-7bb0-4981-85a4-998b8a5026ce-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:52:27.494478 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.494485    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:52:27.494685 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.494494    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fg2\" (UniqueName: \"kubernetes.io/projected/e08cbb18-7bb0-4981-85a4-998b8a5026ce-kube-api-access-56fg2\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:52:27.494685 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:27.494502    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e08cbb18-7bb0-4981-85a4-998b8a5026ce-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:52:28.031842 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.031810    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerID="f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4" exitCode=0
Sep 17 15:52:28.032024 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.031872    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"
Sep 17 15:52:28.032024 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.031874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerDied","Data":"f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4"}
Sep 17 15:52:28.032024 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.032014    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr" event={"ID":"e08cbb18-7bb0-4981-85a4-998b8a5026ce","Type":"ContainerDied","Data":"bf6d4404e99a397d23bd10ca9e0b743bd5e19d6ceeeb8a12bbc59932aa11aff4"}
Sep 17 15:52:28.032131 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.032044    2575 scope.go:117] "RemoveContainer" containerID="f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4"
Sep 17 15:52:28.040382 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.040364    2575 scope.go:117] "RemoveContainer" containerID="8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335"
Sep 17 15:52:28.050342 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.050315    2575 scope.go:117] "RemoveContainer" containerID="a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0"
Sep 17 15:52:28.058210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.058192    2575 scope.go:117] "RemoveContainer" containerID="db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd"
Sep 17 15:52:28.062919 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.062900    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"]
Sep 17 15:52:28.065830 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.065807    2575 scope.go:117] "RemoveContainer" containerID="f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4"
Sep 17 15:52:28.066128 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:52:28.066105    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4\": container with ID starting with f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4 not found: ID does not exist" containerID="f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4"
Sep 17 15:52:28.066189 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066138    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4"} err="failed to get container status \"f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4\": rpc error: code = NotFound desc = could not find container \"f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4\": container with ID starting with f5d45c07ad13b7801c8d7775f23eb13d94ce569bb7043465d1347af5d03f8dd4 not found: ID does not exist"
Sep 17 15:52:28.066189 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066169    2575 scope.go:117] "RemoveContainer" containerID="8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335"
Sep 17 15:52:28.066440 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:52:28.066423    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335\": container with ID starting with 8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335 not found: ID does not exist" containerID="8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335"
Sep 17 15:52:28.066490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066446    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335"} err="failed to get container status \"8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335\": rpc error: code = NotFound desc = could not find container \"8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335\": container with ID starting with 8f6176a9bbb3f93392bf640786ee41547a36bf11e0a9730b23964b35123d5335 not found: ID does not exist"
Sep 17 15:52:28.066490 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066460    2575 scope.go:117] "RemoveContainer" containerID="a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0"
Sep 17 15:52:28.066705 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:52:28.066688    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0\": container with ID starting with a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0 not found: ID does not exist" containerID="a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0"
Sep 17 15:52:28.066747 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066711    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0"} err="failed to get container status \"a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0\": rpc error: code = NotFound desc = could not find container \"a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0\": container with ID starting with a80df66a50ff24971dd5cf8b8d15b81e411f724fd110b45acefaca711cba59c0 not found: ID does not exist"
Sep 17 15:52:28.066747 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066726    2575 scope.go:117] "RemoveContainer" containerID="db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd"
Sep 17 15:52:28.066991 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:52:28.066968    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd\": container with ID starting with db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd not found: ID does not exist" containerID="db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd"
Sep 17 15:52:28.067070 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.066993    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd"} err="failed to get container status \"db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd\": rpc error: code = NotFound desc = could not find container \"db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd\": container with ID starting with db60733783c97af5dea0c21a11698f1efec6bd83743e9de01c84d9f9493e9cdd not found: ID does not exist"
Sep 17 15:52:28.075877 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:28.075854    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-5b828-predictor-5bd8955789-48nmr"]
Sep 17 15:52:29.056400 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:52:29.056368    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" path="/var/lib/kubelet/pods/e08cbb18-7bb0-4981-85a4-998b8a5026ce/volumes"
Sep 17 15:53:22.443144 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:22.443113    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"]
Sep 17 15:53:22.443630 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:22.443407    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container" containerID="cri-o://89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f" gracePeriod=30
Sep 17 15:53:22.443630 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:22.443479    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kube-rbac-proxy" containerID="cri-o://20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13" gracePeriod=30
Sep 17 15:53:23.236019 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:23.235979    2575 generic.go:356] "Generic (PLEG): container finished" podID="5fd86159-01c7-40fd-9ae7-00099204f238" containerID="20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13" exitCode=2
Sep 17 15:53:23.236196 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:23.236044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerDied","Data":"20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13"}
Sep 17 15:53:24.592735 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:24.592696    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 17 15:53:26.679676 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.679654    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:53:26.855384 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855315    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5h22\" (UniqueName: \"kubernetes.io/projected/5fd86159-01c7-40fd-9ae7-00099204f238-kube-api-access-g5h22\") pod \"5fd86159-01c7-40fd-9ae7-00099204f238\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") "
Sep 17 15:53:26.855384 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855358    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-cabundle-cert\") pod \"5fd86159-01c7-40fd-9ae7-00099204f238\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") "
Sep 17 15:53:26.855543 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855397    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-kube-rbac-proxy-sar-config\") pod \"5fd86159-01c7-40fd-9ae7-00099204f238\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") "
Sep 17 15:53:26.855543 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855438    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fd86159-01c7-40fd-9ae7-00099204f238-kserve-provision-location\") pod \"5fd86159-01c7-40fd-9ae7-00099204f238\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") "
Sep 17 15:53:26.855543 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855461    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fd86159-01c7-40fd-9ae7-00099204f238-proxy-tls\") pod \"5fd86159-01c7-40fd-9ae7-00099204f238\" (UID: \"5fd86159-01c7-40fd-9ae7-00099204f238\") "
Sep 17 15:53:26.855720 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855679    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5fd86159-01c7-40fd-9ae7-00099204f238" (UID: "5fd86159-01c7-40fd-9ae7-00099204f238"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:26.855803 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855768    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fd86159-01c7-40fd-9ae7-00099204f238" (UID: "5fd86159-01c7-40fd-9ae7-00099204f238"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:53:26.855803 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.855784    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd86159-01c7-40fd-9ae7-00099204f238-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5fd86159-01c7-40fd-9ae7-00099204f238" (UID: "5fd86159-01c7-40fd-9ae7-00099204f238"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:53:26.857614 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.857587    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd86159-01c7-40fd-9ae7-00099204f238-kube-api-access-g5h22" (OuterVolumeSpecName: "kube-api-access-g5h22") pod "5fd86159-01c7-40fd-9ae7-00099204f238" (UID: "5fd86159-01c7-40fd-9ae7-00099204f238"). InnerVolumeSpecName "kube-api-access-g5h22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:53:26.857712 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.857639    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd86159-01c7-40fd-9ae7-00099204f238-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fd86159-01c7-40fd-9ae7-00099204f238" (UID: "5fd86159-01c7-40fd-9ae7-00099204f238"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:53:26.956424 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.956397    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.956424 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.956421    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fd86159-01c7-40fd-9ae7-00099204f238-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.956564 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.956435    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fd86159-01c7-40fd-9ae7-00099204f238-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.956564 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.956450    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5h22\" (UniqueName: \"kubernetes.io/projected/5fd86159-01c7-40fd-9ae7-00099204f238-kube-api-access-g5h22\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:53:26.956564 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:26.956461    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fd86159-01c7-40fd-9ae7-00099204f238-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:53:27.252651 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.252623    2575 generic.go:356] "Generic (PLEG): container finished" podID="5fd86159-01c7-40fd-9ae7-00099204f238" containerID="89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f" exitCode=0
Sep 17 15:53:27.252780 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.252681    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"
Sep 17 15:53:27.252780 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.252703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerDied","Data":"89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f"}
Sep 17 15:53:27.252780 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.252762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4" event={"ID":"5fd86159-01c7-40fd-9ae7-00099204f238","Type":"ContainerDied","Data":"65104b2d8a59df8063203f08414c078647e2fa890a85c70ecc8993382c958b6c"}
Sep 17 15:53:27.252877 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.252788    2575 scope.go:117] "RemoveContainer" containerID="20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13"
Sep 17 15:53:27.260815 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.260798    2575 scope.go:117] "RemoveContainer" containerID="89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f"
Sep 17 15:53:27.267601 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.267585    2575 scope.go:117] "RemoveContainer" containerID="935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f"
Sep 17 15:53:27.274751 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.274734    2575 scope.go:117] "RemoveContainer" containerID="20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13"
Sep 17 15:53:27.275003 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:53:27.274987    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13\": container with ID starting with 20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13 not found: ID does not exist" containerID="20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13"
Sep 17 15:53:27.275060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.275009    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13"} err="failed to get container status \"20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13\": rpc error: code = NotFound desc = could not find container \"20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13\": container with ID starting with 20412526cb7b6163e7122c72c9db977490c564e703d774e6ded36a84a8f15d13 not found: ID does not exist"
Sep 17 15:53:27.275060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.275026    2575 scope.go:117] "RemoveContainer" containerID="89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f"
Sep 17 15:53:27.275224 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:53:27.275206    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f\": container with ID starting with 89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f not found: ID does not exist" containerID="89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f"
Sep 17 15:53:27.275264 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.275228    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f"} err="failed to get container status \"89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f\": rpc error: code = NotFound desc = could not find container \"89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f\": container with ID starting with 89e3618f62c56047f1f8e420322d37614ac572302716eb3f163918f76e46e09f not found: ID does not exist"
Sep 17 15:53:27.275264 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.275241    2575 scope.go:117] "RemoveContainer" containerID="935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f"
Sep 17 15:53:27.275406 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:53:27.275391    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f\": container with ID starting with 935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f not found: ID does not exist" containerID="935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f"
Sep 17 15:53:27.275447 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.275408    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f"} err="failed to get container status \"935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f\": rpc error: code = NotFound desc = could not find container \"935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f\": container with ID starting with 935f8067bf1d8f826237e8f678596a614bae849074381142518b622fc8e9539f not found: ID does not exist"
Sep 17 15:53:27.282480 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.282457    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"]
Sep 17 15:53:27.290827 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.290804    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-m9mf4"]
Sep 17 15:53:27.619330 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619255    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"]
Sep 17 15:53:27.619595 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619578    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="storage-initializer"
Sep 17 15:53:27.619680 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619599    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="storage-initializer"
Sep 17 15:53:27.619680 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619619    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kube-rbac-proxy"
Sep 17 15:53:27.619680 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619628    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kube-rbac-proxy"
Sep 17 15:53:27.619680 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619644    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container"
Sep 17 15:53:27.619680 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619654    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container"
Sep 17 15:53:27.619680 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619673    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619681    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619698    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="storage-initializer"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619706    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="storage-initializer"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619717    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619725    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619733    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619742    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619811    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kube-rbac-proxy"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619824    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="agent"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619834    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08cbb18-7bb0-4981-85a4-998b8a5026ce" containerName="kserve-container"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619843    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kserve-container"
Sep 17 15:53:27.620001 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.619852    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" containerName="kube-rbac-proxy"
Sep 17 15:53:27.624561 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.624539    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.625885 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.625866    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"]
Sep 17 15:53:27.628274 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.628248    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:53:27.628449 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.628428    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:53:27.628532 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.628515    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 15:53:27.628845 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.628823    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:53:27.629027 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.628920    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 15:53:27.629027 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.628924    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:53:27.629327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.629294    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 15:53:27.630479 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.630453    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.633923 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.633903    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 15:53:27.647383 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.647364    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"]
Sep 17 15:53:27.650588 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.650570    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"]
Sep 17 15:53:27.763192 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rnt\" (UniqueName: \"kubernetes.io/projected/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-api-access-d9rnt\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763196    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763295    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc614e2-0cea-44a5-9b71-60998bbcdf94-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hd22\" (UniqueName: \"kubernetes.io/projected/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-api-access-4hd22\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efc614e2-0cea-44a5-9b71-60998bbcdf94-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.763523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.763431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.863713 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.863820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.863820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc614e2-0cea-44a5-9b71-60998bbcdf94-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.863820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.863820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hd22\" (UniqueName: \"kubernetes.io/projected/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-api-access-4hd22\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.863820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efc614e2-0cea-44a5-9b71-60998bbcdf94-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.863820 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.864147 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.864147 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rnt\" (UniqueName: \"kubernetes.io/projected/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-api-access-d9rnt\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.864147 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.863916    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.864290 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.864233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc614e2-0cea-44a5-9b71-60998bbcdf94-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.864347 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.864281    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.864571 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.864535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.864641 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.864614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.864641 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.864631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.864762 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.864746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.867016 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.867000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.867138 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.867120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efc614e2-0cea-44a5-9b71-60998bbcdf94-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.881467 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.881410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rnt\" (UniqueName: \"kubernetes.io/projected/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-api-access-d9rnt\") pod \"isvc-canary-stop-predictor-556d599d48-dtcl9\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:27.881567 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.881550    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hd22\" (UniqueName: \"kubernetes.io/projected/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-api-access-4hd22\") pod \"isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.940610 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.940586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:27.946138 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:27.946117    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:28.227521 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:28.227475    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"]
Sep 17 15:53:28.230283 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:53:28.230143    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc614e2_0cea_44a5_9b71_60998bbcdf94.slice/crio-76676dd72eda1c95aaf3f97ed7385b593adcf518587d67723fc65c39e6ab1962 WatchSource:0}: Error finding container 76676dd72eda1c95aaf3f97ed7385b593adcf518587d67723fc65c39e6ab1962: Status 404 returned error can't find the container with id 76676dd72eda1c95aaf3f97ed7385b593adcf518587d67723fc65c39e6ab1962
Sep 17 15:53:28.230283 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:28.230203    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"]
Sep 17 15:53:28.232442 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:53:28.232420    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34748b8_07d1_4f97_8a3a_1ea6b1dcf6ad.slice/crio-51f3c955ea34a2d1bf64a37f4882613d7c6de4d060a3b6ea9f57635790af723a WatchSource:0}: Error finding container 51f3c955ea34a2d1bf64a37f4882613d7c6de4d060a3b6ea9f57635790af723a: Status 404 returned error can't find the container with id 51f3c955ea34a2d1bf64a37f4882613d7c6de4d060a3b6ea9f57635790af723a
Sep 17 15:53:28.257832 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:28.257810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerStarted","Data":"76676dd72eda1c95aaf3f97ed7385b593adcf518587d67723fc65c39e6ab1962"}
Sep 17 15:53:28.259156 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:28.259130    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerStarted","Data":"51f3c955ea34a2d1bf64a37f4882613d7c6de4d060a3b6ea9f57635790af723a"}
Sep 17 15:53:29.057972 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:29.057915    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd86159-01c7-40fd-9ae7-00099204f238" path="/var/lib/kubelet/pods/5fd86159-01c7-40fd-9ae7-00099204f238/volumes"
Sep 17 15:53:29.265992 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:29.265938    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerStarted","Data":"2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f"}
Sep 17 15:53:29.267476 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:29.267448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerStarted","Data":"2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7"}
Sep 17 15:53:32.285321 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:32.285206    2575 generic.go:356] "Generic (PLEG): container finished" podID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerID="2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f" exitCode=0
Sep 17 15:53:32.285321 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:32.285235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerDied","Data":"2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f"}
Sep 17 15:53:33.291216 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:33.291182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerStarted","Data":"abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c"}
Sep 17 15:53:33.291694 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:33.291257    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerStarted","Data":"bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13"}
Sep 17 15:53:33.291694 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:33.291486    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:33.292843 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:33.292822    2575 generic.go:356] "Generic (PLEG): container finished" podID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerID="2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7" exitCode=0
Sep 17 15:53:33.292928 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:33.292849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerDied","Data":"2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7"}
Sep 17 15:53:33.321395 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:33.321355    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podStartSLOduration=2.589420409 podStartE2EDuration="6.321340059s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.73191965s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:33.320640049 +0000 UTC m=+860.750803840" watchObservedRunningTime="2026-09-17 15:53:33.321340059 +0000 UTC m=+860.751503839"
Sep 17 15:53:34.300135 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:34.300102    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerStarted","Data":"a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d"}
Sep 17 15:53:34.300487 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:34.300158    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerStarted","Data":"e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097"}
Sep 17 15:53:34.300487 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:34.300404    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:34.300603 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:34.300582    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:34.301578 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:34.301556    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:53:34.332479 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:34.332436    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podStartSLOduration=3.23016706 podStartE2EDuration="7.332422452s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.102255392s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:34.331616291 +0000 UTC m=+861.761780073" watchObservedRunningTime="2026-09-17 15:53:34.332422452 +0000 UTC m=+861.762586298"
Sep 17 15:53:35.308855 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:35.308795    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:53:35.309327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:35.308927    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:35.310032 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:35.310002    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:36.312506 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:36.312465    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:40.314524 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:40.314497    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:53:40.314981 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:40.314936    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:53:41.317100 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:41.317075    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:53:41.317644 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:41.317621    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:47.820886 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.820847    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"]
Sep 17 15:53:47.828586 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.828555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:47.832482 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.832453    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-883e7-kube-rbac-proxy-sar-config\""
Sep 17 15:53:47.832647 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.832614    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-883e7-predictor-serving-cert\""
Sep 17 15:53:47.852584 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.852555    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"]
Sep 17 15:53:47.901360 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.901335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnpf\" (UniqueName: \"kubernetes.io/projected/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-api-access-hvnpf\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:47.901499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.901374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65de7407-f09d-4c1d-b69c-f5cf716404e0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:47.901499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.901411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65de7407-f09d-4c1d-b69c-f5cf716404e0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:47.901499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.901429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:47.901499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:47.901456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.002327 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.002295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnpf\" (UniqueName: \"kubernetes.io/projected/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-api-access-hvnpf\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.002499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.002385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65de7407-f09d-4c1d-b69c-f5cf716404e0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.002499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.002452    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65de7407-f09d-4c1d-b69c-f5cf716404e0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.002499 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.002481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.002660 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.002530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.002855 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.002832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65de7407-f09d-4c1d-b69c-f5cf716404e0-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.003294 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.003271    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.003372 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.003295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.005015 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.004997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65de7407-f09d-4c1d-b69c-f5cf716404e0-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.037788 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.037751    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnpf\" (UniqueName: \"kubernetes.io/projected/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-api-access-hvnpf\") pod \"isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.139496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.139425    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:48.436117 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:48.436041    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"]
Sep 17 15:53:48.439352 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:53:48.439324    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65de7407_f09d_4c1d_b69c_f5cf716404e0.slice/crio-dac703ca7e9d89024ad77a00920df3f31731c575d9b4f636fe53fd74bcde4741 WatchSource:0}: Error finding container dac703ca7e9d89024ad77a00920df3f31731c575d9b4f636fe53fd74bcde4741: Status 404 returned error can't find the container with id dac703ca7e9d89024ad77a00920df3f31731c575d9b4f636fe53fd74bcde4741
Sep 17 15:53:49.366000 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:49.365937    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerStarted","Data":"fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe"}
Sep 17 15:53:49.366362 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:49.366032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerStarted","Data":"dac703ca7e9d89024ad77a00920df3f31731c575d9b4f636fe53fd74bcde4741"}
Sep 17 15:53:50.315110 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:50.315060    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:53:51.318089 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:51.318054    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:53:52.377474 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:52.377441    2575 generic.go:356] "Generic (PLEG): container finished" podID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerID="fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe" exitCode=0
Sep 17 15:53:52.377816 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:52.377493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerDied","Data":"fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe"}
Sep 17 15:53:53.384018 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:53.383981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerStarted","Data":"d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb"}
Sep 17 15:53:53.384454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:53.384056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerStarted","Data":"e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2"}
Sep 17 15:53:53.384454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:53.384335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:53.384454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:53.384384    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:53.385735 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:53.385703    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:53:53.420359 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:53.420314    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podStartSLOduration=2.717192056 podStartE2EDuration="6.420299378s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.703107322s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:53.416870627 +0000 UTC m=+880.847034418" watchObservedRunningTime="2026-09-17 15:53:53.420299378 +0000 UTC m=+880.850463159"
Sep 17 15:53:54.387775 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:54.387731    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:53:59.391872 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:59.391844    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:53:59.392378 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:53:59.392355    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:54:00.315722 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:00.315681    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:54:01.318475 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:01.318434    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:09.392882 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:09.392787    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:54:10.315737 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:10.315691    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:54:11.317966 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:11.317917    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:19.392920 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:19.392883    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:54:20.315394 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:20.315354    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:54:21.317619 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:21.317580    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:29.392589 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:29.392547    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:54:30.315118 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:30.315078    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:54:31.317824 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:31.317780    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:39.393069 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:39.393024    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:54:40.315094 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:40.315058    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:54:41.318498 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:41.318458    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 15:54:49.392766 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:49.392728    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:54:50.316111 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.316082    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:54:50.491326 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.491293    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"]
Sep 17 15:54:50.491693 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.491516    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container" containerID="cri-o://e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097" gracePeriod=30
Sep 17 15:54:50.491693 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.491593    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kube-rbac-proxy" containerID="cri-o://a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d" gracePeriod=30
Sep 17 15:54:50.492275 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.492069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:54:50.609070 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.609006    2575 generic.go:356] "Generic (PLEG): container finished" podID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerID="a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d" exitCode=2
Sep 17 15:54:50.609070 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:50.609053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerDied","Data":"a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d"}
Sep 17 15:54:51.313751 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:51.313711    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 15:54:55.234281 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.234255    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:54:55.333645 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.333554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-rbac-proxy-sar-config\") pod \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") "
Sep 17 15:54:55.333645 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.333591    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-cabundle-cert\") pod \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") "
Sep 17 15:54:55.333645 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.333611    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9rnt\" (UniqueName: \"kubernetes.io/projected/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-api-access-d9rnt\") pod \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") "
Sep 17 15:54:55.333645 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.333637    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kserve-provision-location\") pod \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") "
Sep 17 15:54:55.333995 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.333670    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-proxy-tls\") pod \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\" (UID: \"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad\") "
Sep 17 15:54:55.333995 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.333919    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" (UID: "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:55.334099 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.334017    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" (UID: "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:55.334099 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.334029    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" (UID: "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:55.335982 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.335929    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" (UID: "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:55.336090 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.335980    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-api-access-d9rnt" (OuterVolumeSpecName: "kube-api-access-d9rnt") pod "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" (UID: "c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad"). InnerVolumeSpecName "kube-api-access-d9rnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:55.434496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.434455    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:54:55.434496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.434488    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:54:55.434496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.434499    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rnt\" (UniqueName: \"kubernetes.io/projected/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kube-api-access-d9rnt\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:54:55.434496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.434507    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:54:55.434795 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.434517    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:54:55.633889 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.633728    2575 generic.go:356] "Generic (PLEG): container finished" podID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerID="e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097" exitCode=0
Sep 17 15:54:55.633889 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.633794    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"
Sep 17 15:54:55.633889 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.633816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerDied","Data":"e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097"}
Sep 17 15:54:55.633889 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.633874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9" event={"ID":"c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad","Type":"ContainerDied","Data":"51f3c955ea34a2d1bf64a37f4882613d7c6de4d060a3b6ea9f57635790af723a"}
Sep 17 15:54:55.645443 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.633900    2575 scope.go:117] "RemoveContainer" containerID="a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d"
Sep 17 15:54:55.645443 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.642812    2575 scope.go:117] "RemoveContainer" containerID="e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097"
Sep 17 15:54:55.650790 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.650772    2575 scope.go:117] "RemoveContainer" containerID="2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7"
Sep 17 15:54:55.658339 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.658312    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"]
Sep 17 15:54:55.658551 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.658530    2575 scope.go:117] "RemoveContainer" containerID="a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d"
Sep 17 15:54:55.658850 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:54:55.658830    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d\": container with ID starting with a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d not found: ID does not exist" containerID="a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d"
Sep 17 15:54:55.658925 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.658864    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d"} err="failed to get container status \"a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d\": rpc error: code = NotFound desc = could not find container \"a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d\": container with ID starting with a2f114c14c104393a19a9429499325578c6b46b331f2c81f8fc635e0d5ebe06d not found: ID does not exist"
Sep 17 15:54:55.658925 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.658889    2575 scope.go:117] "RemoveContainer" containerID="e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097"
Sep 17 15:54:55.659295 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:54:55.659277    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097\": container with ID starting with e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097 not found: ID does not exist" containerID="e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097"
Sep 17 15:54:55.659356 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.659301    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097"} err="failed to get container status \"e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097\": rpc error: code = NotFound desc = could not find container \"e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097\": container with ID starting with e54bec8596bb33f79afc2ce4c36a4dee8c97c6490e77526317f3c414ffbf8097 not found: ID does not exist"
Sep 17 15:54:55.659356 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.659315    2575 scope.go:117] "RemoveContainer" containerID="2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7"
Sep 17 15:54:55.659568 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:54:55.659550    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7\": container with ID starting with 2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7 not found: ID does not exist" containerID="2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7"
Sep 17 15:54:55.659629 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.659574    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7"} err="failed to get container status \"2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7\": rpc error: code = NotFound desc = could not find container \"2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7\": container with ID starting with 2e90eb944bdd5d4fde72304aecc03f9c435b40120c0ed11fbbe35f9b708cd9a7 not found: ID does not exist"
Sep 17 15:54:55.665873 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:55.665850    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-dtcl9"]
Sep 17 15:54:57.060428 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:57.060393    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" path="/var/lib/kubelet/pods/c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad/volumes"
Sep 17 15:54:57.494037 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:57.493998    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"]
Sep 17 15:54:57.494323 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:57.494294    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" containerID="cri-o://bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13" gracePeriod=30
Sep 17 15:54:57.494423 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:57.494326    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kube-rbac-proxy" containerID="cri-o://abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c" gracePeriod=30
Sep 17 15:54:57.646097 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:57.646063    2575 generic.go:356] "Generic (PLEG): container finished" podID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerID="abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c" exitCode=2
Sep 17 15:54:57.646253 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:57.646125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerDied","Data":"abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c"}
Sep 17 15:54:59.393110 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:54:59.393072    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:00.309162 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:00.309111    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 17 15:55:00.315496 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:00.315471    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 15:55:02.037632 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.037610    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:55:02.084817 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.084749    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hd22\" (UniqueName: \"kubernetes.io/projected/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-api-access-4hd22\") pod \"efc614e2-0cea-44a5-9b71-60998bbcdf94\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") "
Sep 17 15:55:02.084817 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.084795    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc614e2-0cea-44a5-9b71-60998bbcdf94-kserve-provision-location\") pod \"efc614e2-0cea-44a5-9b71-60998bbcdf94\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") "
Sep 17 15:55:02.084817 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.084813    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efc614e2-0cea-44a5-9b71-60998bbcdf94-proxy-tls\") pod \"efc614e2-0cea-44a5-9b71-60998bbcdf94\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") "
Sep 17 15:55:02.085124 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.084840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-rbac-proxy-sar-config\") pod \"efc614e2-0cea-44a5-9b71-60998bbcdf94\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") "
Sep 17 15:55:02.085124 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.084899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-cabundle-cert\") pod \"efc614e2-0cea-44a5-9b71-60998bbcdf94\" (UID: \"efc614e2-0cea-44a5-9b71-60998bbcdf94\") "
Sep 17 15:55:02.085234 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.085135    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc614e2-0cea-44a5-9b71-60998bbcdf94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efc614e2-0cea-44a5-9b71-60998bbcdf94" (UID: "efc614e2-0cea-44a5-9b71-60998bbcdf94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:02.085234 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.085213    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "efc614e2-0cea-44a5-9b71-60998bbcdf94" (UID: "efc614e2-0cea-44a5-9b71-60998bbcdf94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:02.085346 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.085307    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efc614e2-0cea-44a5-9b71-60998bbcdf94" (UID: "efc614e2-0cea-44a5-9b71-60998bbcdf94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:02.087057 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.087029    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-api-access-4hd22" (OuterVolumeSpecName: "kube-api-access-4hd22") pod "efc614e2-0cea-44a5-9b71-60998bbcdf94" (UID: "efc614e2-0cea-44a5-9b71-60998bbcdf94"). InnerVolumeSpecName "kube-api-access-4hd22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:02.087156 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.087069    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc614e2-0cea-44a5-9b71-60998bbcdf94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "efc614e2-0cea-44a5-9b71-60998bbcdf94" (UID: "efc614e2-0cea-44a5-9b71-60998bbcdf94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:02.185587 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.185546    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efc614e2-0cea-44a5-9b71-60998bbcdf94-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:02.185587 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.185583    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efc614e2-0cea-44a5-9b71-60998bbcdf94-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:02.185587 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.185593    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:02.185771 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.185601    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efc614e2-0cea-44a5-9b71-60998bbcdf94-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:02.185771 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.185611    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hd22\" (UniqueName: \"kubernetes.io/projected/efc614e2-0cea-44a5-9b71-60998bbcdf94-kube-api-access-4hd22\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:02.666549 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.666510    2575 generic.go:356] "Generic (PLEG): container finished" podID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerID="bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13" exitCode=0
Sep 17 15:55:02.666711 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.666565    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"
Sep 17 15:55:02.666711 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.666592    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerDied","Data":"bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13"}
Sep 17 15:55:02.666711 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.666653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj" event={"ID":"efc614e2-0cea-44a5-9b71-60998bbcdf94","Type":"ContainerDied","Data":"76676dd72eda1c95aaf3f97ed7385b593adcf518587d67723fc65c39e6ab1962"}
Sep 17 15:55:02.666711 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.666679    2575 scope.go:117] "RemoveContainer" containerID="abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c"
Sep 17 15:55:02.675461 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.675441    2575 scope.go:117] "RemoveContainer" containerID="bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13"
Sep 17 15:55:02.682625 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.682609    2575 scope.go:117] "RemoveContainer" containerID="2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f"
Sep 17 15:55:02.689544 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.689523    2575 scope.go:117] "RemoveContainer" containerID="abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c"
Sep 17 15:55:02.689900 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:55:02.689873    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c\": container with ID starting with abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c not found: ID does not exist" containerID="abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c"
Sep 17 15:55:02.690015 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.689914    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c"} err="failed to get container status \"abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c\": rpc error: code = NotFound desc = could not find container \"abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c\": container with ID starting with abd338e30cab56202b35c8b191aaf8c8c16efa1a2af29d563118d05361862d8c not found: ID does not exist"
Sep 17 15:55:02.690015 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.689938    2575 scope.go:117] "RemoveContainer" containerID="bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13"
Sep 17 15:55:02.690297 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:55:02.690278    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13\": container with ID starting with bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13 not found: ID does not exist" containerID="bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13"
Sep 17 15:55:02.690378 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.690307    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13"} err="failed to get container status \"bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13\": rpc error: code = NotFound desc = could not find container \"bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13\": container with ID starting with bdacd3a6aa2c7044379615741ce6e262b563e316f6f8b1900fe7d38347fadf13 not found: ID does not exist"
Sep 17 15:55:02.690378 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.690327    2575 scope.go:117] "RemoveContainer" containerID="2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f"
Sep 17 15:55:02.690648 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:55:02.690629    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f\": container with ID starting with 2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f not found: ID does not exist" containerID="2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f"
Sep 17 15:55:02.690728 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.690657    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f"} err="failed to get container status \"2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f\": rpc error: code = NotFound desc = could not find container \"2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f\": container with ID starting with 2b3f2067f7252a811daa2d1823186c9e6b1c48ebb2df8f83128a4a584f64516f not found: ID does not exist"
Sep 17 15:55:02.691693 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.691674    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"]
Sep 17 15:55:02.697817 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:02.697798    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5cbd69c75c-t8hpj"]
Sep 17 15:55:03.056678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:03.056636    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" path="/var/lib/kubelet/pods/efc614e2-0cea-44a5-9b71-60998bbcdf94/volumes"
Sep 17 15:55:07.827703 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827668    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"]
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827934    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827959    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827967    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kube-rbac-proxy"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827973    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kube-rbac-proxy"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827984    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827990    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.827997    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kube-rbac-proxy"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828002    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kube-rbac-proxy"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828020    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="storage-initializer"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828025    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="storage-initializer"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828033    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="storage-initializer"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828039    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="storage-initializer"
Sep 17 15:55:07.828095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828099    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kserve-container"
Sep 17 15:55:07.828466 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828107    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kserve-container"
Sep 17 15:55:07.828466 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828114    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34748b8-07d1-4f97-8a3a-1ea6b1dcf6ad" containerName="kube-rbac-proxy"
Sep 17 15:55:07.828466 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.828121    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc614e2-0cea-44a5-9b71-60998bbcdf94" containerName="kube-rbac-proxy"
Sep 17 15:55:07.832409 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.832392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:07.835037 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.835014    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-e083cc-kube-rbac-proxy-sar-config\""
Sep 17 15:55:07.835289 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.835264    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-e083cc-predictor-serving-cert\""
Sep 17 15:55:07.843671 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.843650    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"]
Sep 17 15:55:07.925215 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.925177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19afa13-c6c3-496f-99fb-051b8e378841-kserve-provision-location\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:07.925215 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.925222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19afa13-c6c3-496f-99fb-051b8e378841-proxy-tls\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:07.925424 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.925258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-kube-rbac-proxy-sar-config\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:07.925424 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.925317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldfcc\" (UniqueName: \"kubernetes.io/projected/e19afa13-c6c3-496f-99fb-051b8e378841-kube-api-access-ldfcc\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:07.925424 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:07.925336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-cabundle-cert\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.026275 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.026243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19afa13-c6c3-496f-99fb-051b8e378841-kserve-provision-location\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.026454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.026282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19afa13-c6c3-496f-99fb-051b8e378841-proxy-tls\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.026454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.026325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-kube-rbac-proxy-sar-config\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.026454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.026382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldfcc\" (UniqueName: \"kubernetes.io/projected/e19afa13-c6c3-496f-99fb-051b8e378841-kube-api-access-ldfcc\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.026454 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.026405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-cabundle-cert\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.026726 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.026701    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19afa13-c6c3-496f-99fb-051b8e378841-kserve-provision-location\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.027054 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.027035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-kube-rbac-proxy-sar-config\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.027127 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.027045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-cabundle-cert\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.028925 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.028908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19afa13-c6c3-496f-99fb-051b8e378841-proxy-tls\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.037408 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.037379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldfcc\" (UniqueName: \"kubernetes.io/projected/e19afa13-c6c3-496f-99fb-051b8e378841-kube-api-access-ldfcc\") pod \"isvc-primary-e083cc-predictor-7c56885448-p7l2n\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") " pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.141462 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.141375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:08.416306 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.416279    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"]
Sep 17 15:55:08.418849 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:55:08.418819    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19afa13_c6c3_496f_99fb_051b8e378841.slice/crio-6880f2332e8a2a77fba2ef7c57f44774df65329ed24eb21f6514025bcae2fc65 WatchSource:0}: Error finding container 6880f2332e8a2a77fba2ef7c57f44774df65329ed24eb21f6514025bcae2fc65: Status 404 returned error can't find the container with id 6880f2332e8a2a77fba2ef7c57f44774df65329ed24eb21f6514025bcae2fc65
Sep 17 15:55:08.692299 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.692215    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerStarted","Data":"3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4"}
Sep 17 15:55:08.692299 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:08.692277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerStarted","Data":"6880f2332e8a2a77fba2ef7c57f44774df65329ed24eb21f6514025bcae2fc65"}
Sep 17 15:55:09.393387 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:09.393358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:55:12.709802 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:12.709768    2575 generic.go:356] "Generic (PLEG): container finished" podID="e19afa13-c6c3-496f-99fb-051b8e378841" containerID="3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4" exitCode=0
Sep 17 15:55:12.710209 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:12.709820    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerDied","Data":"3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4"}
Sep 17 15:55:13.718063 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:13.718030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerStarted","Data":"86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a"}
Sep 17 15:55:13.718472 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:13.718091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerStarted","Data":"b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd"}
Sep 17 15:55:13.718472 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:13.718271    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:13.740468 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:13.740427    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podStartSLOduration=2.7104158480000002 podStartE2EDuration="6.740413627s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.029997779s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:13.739384361 +0000 UTC m=+961.169548142" watchObservedRunningTime="2026-09-17 15:55:13.740413627 +0000 UTC m=+961.170577407"
Sep 17 15:55:14.721667 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:14.721633    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:14.722966 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:14.722925    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:55:15.725647 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:15.725605    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:55:20.730612 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:20.730581    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:55:20.731142 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:20.731115    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:55:30.731689 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:30.731654    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:55:37.943849 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:37.943753    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"]
Sep 17 15:55:37.944291 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:37.944077    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" containerID="cri-o://e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2" gracePeriod=30
Sep 17 15:55:37.944291 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:37.944102    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kube-rbac-proxy" containerID="cri-o://d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb" gracePeriod=30
Sep 17 15:55:38.813787 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:38.813755    2575 generic.go:356] "Generic (PLEG): container finished" podID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerID="d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb" exitCode=2
Sep 17 15:55:38.813943 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:38.813829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerDied","Data":"d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb"}
Sep 17 15:55:39.388761 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:39.388719    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 17 15:55:39.392394 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:39.392372    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 17 15:55:40.731661 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:40.731619    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:55:42.178661 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.178638    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:55:42.285926 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.285853    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65de7407-f09d-4c1d-b69c-f5cf716404e0-kserve-provision-location\") pod \"65de7407-f09d-4c1d-b69c-f5cf716404e0\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") "
Sep 17 15:55:42.285926 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.285908    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvnpf\" (UniqueName: \"kubernetes.io/projected/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-api-access-hvnpf\") pod \"65de7407-f09d-4c1d-b69c-f5cf716404e0\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") "
Sep 17 15:55:42.286139 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.285930    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-rbac-proxy-sar-config\") pod \"65de7407-f09d-4c1d-b69c-f5cf716404e0\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") "
Sep 17 15:55:42.286139 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.285974    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-cabundle-cert\") pod \"65de7407-f09d-4c1d-b69c-f5cf716404e0\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") "
Sep 17 15:55:42.286139 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.286015    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65de7407-f09d-4c1d-b69c-f5cf716404e0-proxy-tls\") pod \"65de7407-f09d-4c1d-b69c-f5cf716404e0\" (UID: \"65de7407-f09d-4c1d-b69c-f5cf716404e0\") "
Sep 17 15:55:42.286294 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.286188    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65de7407-f09d-4c1d-b69c-f5cf716404e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65de7407-f09d-4c1d-b69c-f5cf716404e0" (UID: "65de7407-f09d-4c1d-b69c-f5cf716404e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:55:42.286353 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.286330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65de7407-f09d-4c1d-b69c-f5cf716404e0" (UID: "65de7407-f09d-4c1d-b69c-f5cf716404e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:42.286417 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.286386    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65de7407-f09d-4c1d-b69c-f5cf716404e0" (UID: "65de7407-f09d-4c1d-b69c-f5cf716404e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:42.288023 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.288003    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-api-access-hvnpf" (OuterVolumeSpecName: "kube-api-access-hvnpf") pod "65de7407-f09d-4c1d-b69c-f5cf716404e0" (UID: "65de7407-f09d-4c1d-b69c-f5cf716404e0"). InnerVolumeSpecName "kube-api-access-hvnpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:42.288181 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.288162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65de7407-f09d-4c1d-b69c-f5cf716404e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65de7407-f09d-4c1d-b69c-f5cf716404e0" (UID: "65de7407-f09d-4c1d-b69c-f5cf716404e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:42.387357 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.387335    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65de7407-f09d-4c1d-b69c-f5cf716404e0-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:42.387357 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.387357    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnpf\" (UniqueName: \"kubernetes.io/projected/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-api-access-hvnpf\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:42.387505 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.387366    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:42.387505 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.387376    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65de7407-f09d-4c1d-b69c-f5cf716404e0-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:42.387505 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.387390    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65de7407-f09d-4c1d-b69c-f5cf716404e0-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:55:42.833365 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.833332    2575 generic.go:356] "Generic (PLEG): container finished" podID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerID="e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2" exitCode=0
Sep 17 15:55:42.833523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.833380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerDied","Data":"e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2"}
Sep 17 15:55:42.833523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.833384    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"
Sep 17 15:55:42.833523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.833436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9" event={"ID":"65de7407-f09d-4c1d-b69c-f5cf716404e0","Type":"ContainerDied","Data":"dac703ca7e9d89024ad77a00920df3f31731c575d9b4f636fe53fd74bcde4741"}
Sep 17 15:55:42.833523 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.833466    2575 scope.go:117] "RemoveContainer" containerID="d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb"
Sep 17 15:55:42.842051 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.841930    2575 scope.go:117] "RemoveContainer" containerID="e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2"
Sep 17 15:55:42.849361 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.849344    2575 scope.go:117] "RemoveContainer" containerID="fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe"
Sep 17 15:55:42.856199 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.856183    2575 scope.go:117] "RemoveContainer" containerID="d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb"
Sep 17 15:55:42.856454 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:55:42.856433    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb\": container with ID starting with d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb not found: ID does not exist" containerID="d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb"
Sep 17 15:55:42.856526 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.856466    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb"} err="failed to get container status \"d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb\": rpc error: code = NotFound desc = could not find container \"d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb\": container with ID starting with d5e87b666387fbf08656c5c12e2e07ae6d11ca0de8bd7e0f9fb33b62fa05a7fb not found: ID does not exist"
Sep 17 15:55:42.856526 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.856490    2575 scope.go:117] "RemoveContainer" containerID="e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2"
Sep 17 15:55:42.856795 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:55:42.856740    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2\": container with ID starting with e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2 not found: ID does not exist" containerID="e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2"
Sep 17 15:55:42.856795 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.856767    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2"} err="failed to get container status \"e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2\": rpc error: code = NotFound desc = could not find container \"e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2\": container with ID starting with e97e07bc305484b162776fc9f9fa0cc5b68ea1f5faabf069f8e4d8592504c5b2 not found: ID does not exist"
Sep 17 15:55:42.856795 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.856781    2575 scope.go:117] "RemoveContainer" containerID="fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe"
Sep 17 15:55:42.857944 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:55:42.857463    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe\": container with ID starting with fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe not found: ID does not exist" containerID="fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe"
Sep 17 15:55:42.857944 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.857491    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe"} err="failed to get container status \"fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe\": rpc error: code = NotFound desc = could not find container \"fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe\": container with ID starting with fd81fc24ed81d463d404320b58a6b244302994530e9d54c744f5e0f5221008fe not found: ID does not exist"
Sep 17 15:55:42.859000 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.858977    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"]
Sep 17 15:55:42.864391 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:42.864372    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-883e7-predictor-7c9899787c-tqzc9"]
Sep 17 15:55:43.057091 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:43.057056    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" path="/var/lib/kubelet/pods/65de7407-f09d-4c1d-b69c-f5cf716404e0/volumes"
Sep 17 15:55:50.731120 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:50.731082    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:55:58.217042 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217007    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"]
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217316    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container"
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217329    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container"
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217339    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="storage-initializer"
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217344    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="storage-initializer"
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217352    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kube-rbac-proxy"
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217358    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kube-rbac-proxy"
Sep 17 15:55:58.217402 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217405    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kserve-container"
Sep 17 15:55:58.217608 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.217414    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65de7407-f09d-4c1d-b69c-f5cf716404e0" containerName="kube-rbac-proxy"
Sep 17 15:55:58.222170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.222152    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.227845 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.227821    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-3f9cc-predictor-serving-cert\""
Sep 17 15:55:58.228216 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.228200    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-3f9cc-kube-rbac-proxy-sar-config\""
Sep 17 15:55:58.248974 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.248935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"]
Sep 17 15:55:58.404528 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.404500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kserve-provision-location\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.404687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.404535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724t9\" (UniqueName: \"kubernetes.io/projected/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-api-access-724t9\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.404687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.404553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-cabundle-cert\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.404687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.404647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.404827 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.404718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-proxy-tls\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.505527 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.505502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-proxy-tls\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.505687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.505558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kserve-provision-location\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.505687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.505592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724t9\" (UniqueName: \"kubernetes.io/projected/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-api-access-724t9\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.505687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.505613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-cabundle-cert\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.505687 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.505655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.506090 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.506069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kserve-provision-location\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.506363 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.506343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.506422 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.506389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-cabundle-cert\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.508087 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.508069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-proxy-tls\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.522449 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.522430    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-724t9\" (UniqueName: \"kubernetes.io/projected/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-api-access-724t9\") pod \"isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.532270 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.532251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:55:58.819300 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.819252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"]
Sep 17 15:55:58.821461 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:55:58.821430    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f90e7d6_0bc4_4590_b5d1_5ada2a3dccb3.slice/crio-d04e6ed9586b22c6827bff4a8f81720aa2b76bb16f669aa283d4accd13f7f434 WatchSource:0}: Error finding container d04e6ed9586b22c6827bff4a8f81720aa2b76bb16f669aa283d4accd13f7f434: Status 404 returned error can't find the container with id d04e6ed9586b22c6827bff4a8f81720aa2b76bb16f669aa283d4accd13f7f434
Sep 17 15:55:58.893707 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.893675    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerStarted","Data":"6f77c3ad6ce8f22e6ac615ad84462c3c91768a980bfd3f55a5bb926adf31b5ac"}
Sep 17 15:55:58.893817 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:55:58.893748    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerStarted","Data":"d04e6ed9586b22c6827bff4a8f81720aa2b76bb16f669aa283d4accd13f7f434"}
Sep 17 15:56:00.731443 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:00.731410    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:56:02.910445 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:02.910360    2575 generic.go:356] "Generic (PLEG): container finished" podID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerID="6f77c3ad6ce8f22e6ac615ad84462c3c91768a980bfd3f55a5bb926adf31b5ac" exitCode=0
Sep 17 15:56:02.910850 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:02.910441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerDied","Data":"6f77c3ad6ce8f22e6ac615ad84462c3c91768a980bfd3f55a5bb926adf31b5ac"}
Sep 17 15:56:03.917085 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:03.917051    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerStarted","Data":"3a3d2aacfe3bbb240ca38d41af3550aa44b386d53b699b84835cd91e8046687c"}
Sep 17 15:56:03.917495 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:03.917117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerStarted","Data":"ee91630473e047450fcf8dbc3d460b81ead2d93186cadaddea2fa56dbc0fb4e2"}
Sep 17 15:56:03.917495 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:03.917141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerStarted","Data":"7e8f12e33c35ff2845cd63b1360de1aabd0a149cbfd01302b322d77b4929181f"}
Sep 17 15:56:03.917495 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:03.917335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:56:03.947033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:03.946985    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podStartSLOduration=2.17472612 podStartE2EDuration="5.946970988s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.772244868s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:03.946794286 +0000 UTC m=+1011.376958067" watchObservedRunningTime="2026-09-17 15:56:03.946970988 +0000 UTC m=+1011.377134766"
Sep 17 15:56:04.921044 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:04.921012    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:56:04.921403 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:04.921109    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:56:04.922385 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:04.922356    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:04.923003 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:04.922978    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:05.924531 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:05.924484    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:05.925026 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:05.925006    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:10.731422 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:10.731386    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:56:10.929553 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:10.929521    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:56:10.930134 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:10.930106    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:10.930503 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:10.930474    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:20.731138 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:20.731101    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 15:56:20.930516 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:20.930478    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:20.930973 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:20.930919    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:30.732137 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:30.732101    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:56:30.930904 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:30.930857    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:30.931411 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:30.931387    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:40.930174 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:40.930130    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:40.930644 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:40.930619    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:50.931085 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:50.931035    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:56:50.931563 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:56:50.931517    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:57:00.930506 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:00.930466    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:57:00.931021 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:00.930994    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:57:10.930855 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:10.930693    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:57:10.931450 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:10.931174    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:57:11.210008 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:11.209923    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"]
Sep 17 15:57:11.210201 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:11.210174    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container" containerID="cri-o://b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd" gracePeriod=30
Sep 17 15:57:11.210263 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:11.210204    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kube-rbac-proxy" containerID="cri-o://86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a" gracePeriod=30
Sep 17 15:57:12.179033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:12.178999    2575 generic.go:356] "Generic (PLEG): container finished" podID="e19afa13-c6c3-496f-99fb-051b8e378841" containerID="86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a" exitCode=2
Sep 17 15:57:12.179033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:12.179035    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerDied","Data":"86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a"}
Sep 17 15:57:15.726605 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.726572    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 15:57:15.850684 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.850655    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:57:15.975487 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975403    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldfcc\" (UniqueName: \"kubernetes.io/projected/e19afa13-c6c3-496f-99fb-051b8e378841-kube-api-access-ldfcc\") pod \"e19afa13-c6c3-496f-99fb-051b8e378841\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") "
Sep 17 15:57:15.975487 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975440    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-kube-rbac-proxy-sar-config\") pod \"e19afa13-c6c3-496f-99fb-051b8e378841\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") "
Sep 17 15:57:15.975692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975573    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19afa13-c6c3-496f-99fb-051b8e378841-kserve-provision-location\") pod \"e19afa13-c6c3-496f-99fb-051b8e378841\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") "
Sep 17 15:57:15.975692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975629    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19afa13-c6c3-496f-99fb-051b8e378841-proxy-tls\") pod \"e19afa13-c6c3-496f-99fb-051b8e378841\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") "
Sep 17 15:57:15.975692 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975658    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-cabundle-cert\") pod \"e19afa13-c6c3-496f-99fb-051b8e378841\" (UID: \"e19afa13-c6c3-496f-99fb-051b8e378841\") "
Sep 17 15:57:15.975907 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975886    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e19afa13-c6c3-496f-99fb-051b8e378841" (UID: "e19afa13-c6c3-496f-99fb-051b8e378841"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:15.976000 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.975899    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19afa13-c6c3-496f-99fb-051b8e378841-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e19afa13-c6c3-496f-99fb-051b8e378841" (UID: "e19afa13-c6c3-496f-99fb-051b8e378841"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:15.976130 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.976096    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e19afa13-c6c3-496f-99fb-051b8e378841" (UID: "e19afa13-c6c3-496f-99fb-051b8e378841"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:15.977696 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.977674    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19afa13-c6c3-496f-99fb-051b8e378841-kube-api-access-ldfcc" (OuterVolumeSpecName: "kube-api-access-ldfcc") pod "e19afa13-c6c3-496f-99fb-051b8e378841" (UID: "e19afa13-c6c3-496f-99fb-051b8e378841"). InnerVolumeSpecName "kube-api-access-ldfcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:15.977788 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:15.977746    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19afa13-c6c3-496f-99fb-051b8e378841-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e19afa13-c6c3-496f-99fb-051b8e378841" (UID: "e19afa13-c6c3-496f-99fb-051b8e378841"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:16.076536 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.076500    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e19afa13-c6c3-496f-99fb-051b8e378841-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:16.076536 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.076532    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e19afa13-c6c3-496f-99fb-051b8e378841-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:16.076536 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.076544    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:16.076754 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.076553    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldfcc\" (UniqueName: \"kubernetes.io/projected/e19afa13-c6c3-496f-99fb-051b8e378841-kube-api-access-ldfcc\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:16.076754 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.076563    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e19afa13-c6c3-496f-99fb-051b8e378841-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:16.197037 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.197003    2575 generic.go:356] "Generic (PLEG): container finished" podID="e19afa13-c6c3-496f-99fb-051b8e378841" containerID="b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd" exitCode=0
Sep 17 15:57:16.197186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.197055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerDied","Data":"b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd"}
Sep 17 15:57:16.197186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.197061    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"
Sep 17 15:57:16.197186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.197104    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n" event={"ID":"e19afa13-c6c3-496f-99fb-051b8e378841","Type":"ContainerDied","Data":"6880f2332e8a2a77fba2ef7c57f44774df65329ed24eb21f6514025bcae2fc65"}
Sep 17 15:57:16.197186 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.197130    2575 scope.go:117] "RemoveContainer" containerID="86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a"
Sep 17 15:57:16.206436 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.206419    2575 scope.go:117] "RemoveContainer" containerID="b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd"
Sep 17 15:57:16.214016 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.213998    2575 scope.go:117] "RemoveContainer" containerID="3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4"
Sep 17 15:57:16.221847 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.221822    2575 scope.go:117] "RemoveContainer" containerID="86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a"
Sep 17 15:57:16.222154 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.222135    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"]
Sep 17 15:57:16.222207 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:16.222174    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a\": container with ID starting with 86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a not found: ID does not exist" containerID="86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a"
Sep 17 15:57:16.222252 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.222199    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a"} err="failed to get container status \"86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a\": rpc error: code = NotFound desc = could not find container \"86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a\": container with ID starting with 86ee3d338729133b85c4a294becad5eba517f8ce189a5d24360cd2c227a9fc6a not found: ID does not exist"
Sep 17 15:57:16.222252 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.222220    2575 scope.go:117] "RemoveContainer" containerID="b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd"
Sep 17 15:57:16.222512 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:16.222495    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd\": container with ID starting with b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd not found: ID does not exist" containerID="b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd"
Sep 17 15:57:16.222565 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.222517    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd"} err="failed to get container status \"b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd\": rpc error: code = NotFound desc = could not find container \"b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd\": container with ID starting with b689eeed13df288f5be9f5b8c2c70350ad2e1f3aa7bda6e2295b94989bdaf3dd not found: ID does not exist"
Sep 17 15:57:16.222565 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.222536    2575 scope.go:117] "RemoveContainer" containerID="3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4"
Sep 17 15:57:16.222777 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:16.222761    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4\": container with ID starting with 3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4 not found: ID does not exist" containerID="3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4"
Sep 17 15:57:16.222819 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.222781    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4"} err="failed to get container status \"3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4\": rpc error: code = NotFound desc = could not find container \"3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4\": container with ID starting with 3e2f8d27b6f1e4e82982a541c55cc5981545eb3305aecf39427ef19ec3904ba4 not found: ID does not exist"
Sep 17 15:57:16.228427 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.228373    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e083cc-predictor-7c56885448-p7l2n"]
Sep 17 15:57:16.404681 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.404649    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"]
Sep 17 15:57:16.404989 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.404975    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container"
Sep 17 15:57:16.405057 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.404990    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container"
Sep 17 15:57:16.405057 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.405014    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kube-rbac-proxy"
Sep 17 15:57:16.405057 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.405023    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kube-rbac-proxy"
Sep 17 15:57:16.405057 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.405031    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="storage-initializer"
Sep 17 15:57:16.405057 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.405038    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="storage-initializer"
Sep 17 15:57:16.405301 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.405090    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kserve-container"
Sep 17 15:57:16.405301 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.405098    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" containerName="kube-rbac-proxy"
Sep 17 15:57:16.409504 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.409487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.411882 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.411855    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c41e99-predictor-serving-cert\""
Sep 17 15:57:16.412038 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.411876    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c41e99-kube-rbac-proxy-sar-config\""
Sep 17 15:57:16.412281 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.412267    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-c41e99-dockercfg-k5lhh\""
Sep 17 15:57:16.412281 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.412276    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-c41e99\""
Sep 17 15:57:16.423161 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.423133    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"]
Sep 17 15:57:16.580227 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.580190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ljq\" (UniqueName: \"kubernetes.io/projected/0491273b-35da-4f2a-aa97-a39d094204e8-kube-api-access-q2ljq\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.580412 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.580243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.580412 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.580348    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.580412 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.580396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0491273b-35da-4f2a-aa97-a39d094204e8-kserve-provision-location\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.580522 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.580441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-cabundle-cert\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681101 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ljq\" (UniqueName: \"kubernetes.io/projected/0491273b-35da-4f2a-aa97-a39d094204e8-kube-api-access-q2ljq\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681203 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681203 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681303 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0491273b-35da-4f2a-aa97-a39d094204e8-kserve-provision-location\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681303 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-cabundle-cert\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681588 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0491273b-35da-4f2a-aa97-a39d094204e8-kserve-provision-location\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681860 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.681941 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.681915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-cabundle-cert\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.683771 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.683750    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.693352 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.693332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ljq\" (UniqueName: \"kubernetes.io/projected/0491273b-35da-4f2a-aa97-a39d094204e8-kube-api-access-q2ljq\") pod \"isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") " pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:16.722048 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:16.722027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:17.000522 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:17.000496    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"]
Sep 17 15:57:17.002987 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:57:17.002942    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0491273b_35da_4f2a_aa97_a39d094204e8.slice/crio-9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098 WatchSource:0}: Error finding container 9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098: Status 404 returned error can't find the container with id 9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098
Sep 17 15:57:17.057264 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:17.057241    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19afa13-c6c3-496f-99fb-051b8e378841" path="/var/lib/kubelet/pods/e19afa13-c6c3-496f-99fb-051b8e378841/volumes"
Sep 17 15:57:17.202670 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:17.202635    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" event={"ID":"0491273b-35da-4f2a-aa97-a39d094204e8","Type":"ContainerStarted","Data":"a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844"}
Sep 17 15:57:17.202857 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:17.202697    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" event={"ID":"0491273b-35da-4f2a-aa97-a39d094204e8","Type":"ContainerStarted","Data":"9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098"}
Sep 17 15:57:20.053041 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:20.053010    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:57:20.053377 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:20.053110    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:57:21.220285 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:21.220257    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/0.log"
Sep 17 15:57:21.220744 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:21.220294    2575 generic.go:356] "Generic (PLEG): container finished" podID="0491273b-35da-4f2a-aa97-a39d094204e8" containerID="a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844" exitCode=1
Sep 17 15:57:21.220744 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:21.220385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" event={"ID":"0491273b-35da-4f2a-aa97-a39d094204e8","Type":"ContainerDied","Data":"a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844"}
Sep 17 15:57:22.226095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:22.226065    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/0.log"
Sep 17 15:57:22.226572 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:22.226191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" event={"ID":"0491273b-35da-4f2a-aa97-a39d094204e8","Type":"ContainerStarted","Data":"e98766f9d615f471714b3f8ac51c54d91eaf9f3f536138e9d27210612f574c91"}
Sep 17 15:57:26.242908 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.242882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/1.log"
Sep 17 15:57:26.243322 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.243268    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/0.log"
Sep 17 15:57:26.243322 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.243300    2575 generic.go:356] "Generic (PLEG): container finished" podID="0491273b-35da-4f2a-aa97-a39d094204e8" containerID="e98766f9d615f471714b3f8ac51c54d91eaf9f3f536138e9d27210612f574c91" exitCode=1
Sep 17 15:57:26.243416 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.243349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" event={"ID":"0491273b-35da-4f2a-aa97-a39d094204e8","Type":"ContainerDied","Data":"e98766f9d615f471714b3f8ac51c54d91eaf9f3f536138e9d27210612f574c91"}
Sep 17 15:57:26.243416 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.243402    2575 scope.go:117] "RemoveContainer" containerID="a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844"
Sep 17 15:57:26.243770 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.243751    2575 scope.go:117] "RemoveContainer" containerID="a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844"
Sep 17 15:57:26.255937 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:26.255880    2575 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_kserve-ci-e2e-test_0491273b-35da-4f2a-aa97-a39d094204e8_0 in pod sandbox 9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098 from index: no such id: 'a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844'" containerID="a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844"
Sep 17 15:57:26.256040 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.255927    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_kserve-ci-e2e-test_0491273b-35da-4f2a-aa97-a39d094204e8_0 in pod sandbox 9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098 from index: no such id: 'a6ae27d0667ef4ae5b7a177c98ad12c66fae853833df8c0ff0d9084da3d10844'"
Sep 17 15:57:26.256099 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:26.256073    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_kserve-ci-e2e-test(0491273b-35da-4f2a-aa97-a39d094204e8)\"" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" podUID="0491273b-35da-4f2a-aa97-a39d094204e8"
Sep 17 15:57:26.350222 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:26.350188    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"]
Sep 17 15:57:26.361014 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:26.360981    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-serving-cert: secret "isvc-init-fail-c41e99-predictor-serving-cert" not found
Sep 17 15:57:26.361175 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:26.361072    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls podName:0491273b-35da-4f2a-aa97-a39d094204e8 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:26.861050966 +0000 UTC m=+1094.291214728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls") pod "isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8") : secret "isvc-init-fail-c41e99-predictor-serving-cert" not found
Sep 17 15:57:26.864702 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:26.864671    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-serving-cert: secret "isvc-init-fail-c41e99-predictor-serving-cert" not found
Sep 17 15:57:26.864858 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:57:26.864747    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls podName:0491273b-35da-4f2a-aa97-a39d094204e8 nodeName:}" failed. No retries permitted until 2026-09-17 15:57:27.864733127 +0000 UTC m=+1095.294896887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls") pod "isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8") : secret "isvc-init-fail-c41e99-predictor-serving-cert" not found
Sep 17 15:57:27.248830 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.248802    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/1.log"
Sep 17 15:57:27.375564 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.375538    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/1.log"
Sep 17 15:57:27.378754 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.378739    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:27.469913 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.469883    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls\") pod \"0491273b-35da-4f2a-aa97-a39d094204e8\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") "
Sep 17 15:57:27.470085 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.469936    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-kube-rbac-proxy-sar-config\") pod \"0491273b-35da-4f2a-aa97-a39d094204e8\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") "
Sep 17 15:57:27.470085 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.469991    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2ljq\" (UniqueName: \"kubernetes.io/projected/0491273b-35da-4f2a-aa97-a39d094204e8-kube-api-access-q2ljq\") pod \"0491273b-35da-4f2a-aa97-a39d094204e8\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") "
Sep 17 15:57:27.470206 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.470090    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0491273b-35da-4f2a-aa97-a39d094204e8-kserve-provision-location\") pod \"0491273b-35da-4f2a-aa97-a39d094204e8\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") "
Sep 17 15:57:27.470206 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.470142    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-cabundle-cert\") pod \"0491273b-35da-4f2a-aa97-a39d094204e8\" (UID: \"0491273b-35da-4f2a-aa97-a39d094204e8\") "
Sep 17 15:57:27.470363 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.470338    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0491273b-35da-4f2a-aa97-a39d094204e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0491273b-35da-4f2a-aa97-a39d094204e8" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:27.470363 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.470356    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0491273b-35da-4f2a-aa97-a39d094204e8" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:27.470510 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.470490    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0491273b-35da-4f2a-aa97-a39d094204e8" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:27.472236 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.472209    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0491273b-35da-4f2a-aa97-a39d094204e8-kube-api-access-q2ljq" (OuterVolumeSpecName: "kube-api-access-q2ljq") pod "0491273b-35da-4f2a-aa97-a39d094204e8" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8"). InnerVolumeSpecName "kube-api-access-q2ljq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:27.472325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.472297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0491273b-35da-4f2a-aa97-a39d094204e8" (UID: "0491273b-35da-4f2a-aa97-a39d094204e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:27.571060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.570976    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:27.571060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.571013    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ljq\" (UniqueName: \"kubernetes.io/projected/0491273b-35da-4f2a-aa97-a39d094204e8-kube-api-access-q2ljq\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:27.571060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.571023    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0491273b-35da-4f2a-aa97-a39d094204e8-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:27.571060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.571032    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0491273b-35da-4f2a-aa97-a39d094204e8-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:27.571060 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:27.571041    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0491273b-35da-4f2a-aa97-a39d094204e8-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:57:28.254260 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:28.254230    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9_0491273b-35da-4f2a-aa97-a39d094204e8/storage-initializer/1.log"
Sep 17 15:57:28.254640 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:28.254314    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9" event={"ID":"0491273b-35da-4f2a-aa97-a39d094204e8","Type":"ContainerDied","Data":"9843272247ec162277c328d9e2cd47ae6aeb778f34d7fd27156927b49b437098"}
Sep 17 15:57:28.254640 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:28.254339    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"
Sep 17 15:57:28.254640 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:28.254385    2575 scope.go:117] "RemoveContainer" containerID="e98766f9d615f471714b3f8ac51c54d91eaf9f3f536138e9d27210612f574c91"
Sep 17 15:57:28.300427 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:28.300396    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"]
Sep 17 15:57:28.304071 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:28.304046    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c41e99-predictor-5ff8c789c5-xp5n9"]
Sep 17 15:57:29.057392 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:29.057352    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" path="/var/lib/kubelet/pods/0491273b-35da-4f2a-aa97-a39d094204e8/volumes"
Sep 17 15:57:31.604872 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.604825    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"]
Sep 17 15:57:31.605337 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.605140    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" containerName="storage-initializer"
Sep 17 15:57:31.605337 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.605152    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" containerName="storage-initializer"
Sep 17 15:57:31.605337 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.605161    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" containerName="storage-initializer"
Sep 17 15:57:31.605337 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.605168    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" containerName="storage-initializer"
Sep 17 15:57:31.605337 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.605212    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" containerName="storage-initializer"
Sep 17 15:57:31.605337 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.605221    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0491273b-35da-4f2a-aa97-a39d094204e8" containerName="storage-initializer"
Sep 17 15:57:31.609808 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.609792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.612217 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.612199    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-eef9f-kube-rbac-proxy-sar-config\""
Sep 17 15:57:31.612321 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.612205    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-eef9f-predictor-serving-cert\""
Sep 17 15:57:31.619033 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.619013    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"]
Sep 17 15:57:31.803336 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.803305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-cabundle-cert\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.803336 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.803338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/320003f2-ac95-45c9-9651-5313908c018b-kserve-provision-location\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.803512 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.803365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkjz\" (UniqueName: \"kubernetes.io/projected/320003f2-ac95-45c9-9651-5313908c018b-kube-api-access-ljkjz\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.803512 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.803439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/320003f2-ac95-45c9-9651-5313908c018b-proxy-tls\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.803512 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.803492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.909500 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.909408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.909500 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.909458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-cabundle-cert\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.909755 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.909734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/320003f2-ac95-45c9-9651-5313908c018b-kserve-provision-location\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.909936 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.909917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkjz\" (UniqueName: \"kubernetes.io/projected/320003f2-ac95-45c9-9651-5313908c018b-kube-api-access-ljkjz\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.910028 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.909995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/320003f2-ac95-45c9-9651-5313908c018b-proxy-tls\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.910204 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.910187    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/320003f2-ac95-45c9-9651-5313908c018b-kserve-provision-location\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.910459 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.910439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-cabundle-cert\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.910513 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.910453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.912494 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.912475    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/320003f2-ac95-45c9-9651-5313908c018b-proxy-tls\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.920849 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.920830    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkjz\" (UniqueName: \"kubernetes.io/projected/320003f2-ac95-45c9-9651-5313908c018b-kube-api-access-ljkjz\") pod \"raw-sklearn-eef9f-predictor-658d599877-949jb\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") " pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:31.920962 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:31.920939    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:32.197447 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:32.197360    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"]
Sep 17 15:57:32.200357 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:57:32.200332    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320003f2_ac95_45c9_9651_5313908c018b.slice/crio-98f2b35971bee06e169d20cba9308f51dac412a0af4818fc36422423cbc6faab WatchSource:0}: Error finding container 98f2b35971bee06e169d20cba9308f51dac412a0af4818fc36422423cbc6faab: Status 404 returned error can't find the container with id 98f2b35971bee06e169d20cba9308f51dac412a0af4818fc36422423cbc6faab
Sep 17 15:57:32.272019 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:32.271989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerStarted","Data":"a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47"}
Sep 17 15:57:32.272149 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:32.272055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerStarted","Data":"98f2b35971bee06e169d20cba9308f51dac412a0af4818fc36422423cbc6faab"}
Sep 17 15:57:33.310819 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:33.310783    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"]
Sep 17 15:57:33.311252 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:33.311035    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" containerID="cri-o://7e8f12e33c35ff2845cd63b1360de1aabd0a149cbfd01302b322d77b4929181f" gracePeriod=30
Sep 17 15:57:33.311252 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:33.311070    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" containerID="cri-o://3a3d2aacfe3bbb240ca38d41af3550aa44b386d53b699b84835cd91e8046687c" gracePeriod=30
Sep 17 15:57:33.311252 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:33.311073    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" containerID="cri-o://ee91630473e047450fcf8dbc3d460b81ead2d93186cadaddea2fa56dbc0fb4e2" gracePeriod=30
Sep 17 15:57:34.281805 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:34.281771    2575 generic.go:356] "Generic (PLEG): container finished" podID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerID="ee91630473e047450fcf8dbc3d460b81ead2d93186cadaddea2fa56dbc0fb4e2" exitCode=2
Sep 17 15:57:34.282004 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:34.281850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerDied","Data":"ee91630473e047450fcf8dbc3d460b81ead2d93186cadaddea2fa56dbc0fb4e2"}
Sep 17 15:57:35.925642 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:35.925602    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 15:57:36.291208 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:36.291176    2575 generic.go:356] "Generic (PLEG): container finished" podID="320003f2-ac95-45c9-9651-5313908c018b" containerID="a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47" exitCode=0
Sep 17 15:57:36.291352 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:36.291252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerDied","Data":"a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47"}
Sep 17 15:57:37.297533 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:37.297501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerStarted","Data":"7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87"}
Sep 17 15:57:37.297907 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:37.297557    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerStarted","Data":"692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7"}
Sep 17 15:57:37.297907 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:37.297741    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:37.325525 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:37.325466    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podStartSLOduration=2.37604683 podStartE2EDuration="6.325450245s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.949403415s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:37.324275914 +0000 UTC m=+1104.754439710" watchObservedRunningTime="2026-09-17 15:57:37.325450245 +0000 UTC m=+1104.755614082"
Sep 17 15:57:38.303671 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:38.303633    2575 generic.go:356] "Generic (PLEG): container finished" podID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerID="7e8f12e33c35ff2845cd63b1360de1aabd0a149cbfd01302b322d77b4929181f" exitCode=0
Sep 17 15:57:38.304079 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:38.303717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerDied","Data":"7e8f12e33c35ff2845cd63b1360de1aabd0a149cbfd01302b322d77b4929181f"}
Sep 17 15:57:38.304079 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:38.303997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:38.305389 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:38.305361    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:57:39.307230 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:39.307185    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:57:40.052847 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:40.052803    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:57:40.053189 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:40.053170    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:57:40.925407 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:40.925365    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 15:57:44.311938 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:44.311908    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:57:44.312503 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:44.312478    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:57:45.925613 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:45.925570    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 15:57:45.926004 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:45.925699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:57:50.052587 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:50.052547    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:57:50.053005 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:50.052894    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:57:50.925621 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:50.925573    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 15:57:54.313071 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:54.313033    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:57:55.925171 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:57:55.925123    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 15:58:00.053147 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:00.053102    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 15:58:00.053532 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:00.053248    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:58:00.053532 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:00.053447    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:58:00.053608 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:00.053557    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:58:00.924918 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:00.924869    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 15:58:03.399925 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.399893    2575 generic.go:356] "Generic (PLEG): container finished" podID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerID="3a3d2aacfe3bbb240ca38d41af3550aa44b386d53b699b84835cd91e8046687c" exitCode=0
Sep 17 15:58:03.400276 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.399980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerDied","Data":"3a3d2aacfe3bbb240ca38d41af3550aa44b386d53b699b84835cd91e8046687c"}
Sep 17 15:58:03.456917 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.456892    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:58:03.549497 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.549468    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-724t9\" (UniqueName: \"kubernetes.io/projected/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-api-access-724t9\") pod \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") "
Sep 17 15:58:03.549678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.549508    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kserve-provision-location\") pod \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") "
Sep 17 15:58:03.549678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.549570    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-rbac-proxy-sar-config\") pod \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") "
Sep 17 15:58:03.549678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.549625    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-proxy-tls\") pod \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") "
Sep 17 15:58:03.549678 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.549653    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-cabundle-cert\") pod \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\" (UID: \"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3\") "
Sep 17 15:58:03.549910 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.549864    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" (UID: "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:03.550067 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.550040    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" (UID: "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:03.550136 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.550094    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" (UID: "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:03.551788 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.551765    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-api-access-724t9" (OuterVolumeSpecName: "kube-api-access-724t9") pod "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" (UID: "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3"). InnerVolumeSpecName "kube-api-access-724t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:03.551788 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.551773    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" (UID: "8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:03.650611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.650509    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.650611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.650556    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.650611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.650567    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-724t9\" (UniqueName: \"kubernetes.io/projected/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-api-access-724t9\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.650611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.650577    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.650611 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:03.650585    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:04.313020 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.312980    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:58:04.406034 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.405998    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j" event={"ID":"8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3","Type":"ContainerDied","Data":"d04e6ed9586b22c6827bff4a8f81720aa2b76bb16f669aa283d4accd13f7f434"}
Sep 17 15:58:04.406034 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.406034    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"
Sep 17 15:58:04.406514 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.406072    2575 scope.go:117] "RemoveContainer" containerID="3a3d2aacfe3bbb240ca38d41af3550aa44b386d53b699b84835cd91e8046687c"
Sep 17 15:58:04.414335 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.414319    2575 scope.go:117] "RemoveContainer" containerID="ee91630473e047450fcf8dbc3d460b81ead2d93186cadaddea2fa56dbc0fb4e2"
Sep 17 15:58:04.421423 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.421405    2575 scope.go:117] "RemoveContainer" containerID="7e8f12e33c35ff2845cd63b1360de1aabd0a149cbfd01302b322d77b4929181f"
Sep 17 15:58:04.429859 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.429827    2575 scope.go:117] "RemoveContainer" containerID="6f77c3ad6ce8f22e6ac615ad84462c3c91768a980bfd3f55a5bb926adf31b5ac"
Sep 17 15:58:04.431279 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.431260    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"]
Sep 17 15:58:04.433786 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:04.433766    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f9cc-predictor-bd8c66c78-82n6j"]
Sep 17 15:58:05.056908 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:05.056877    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" path="/var/lib/kubelet/pods/8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3/volumes"
Sep 17 15:58:14.313128 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:14.313088    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:58:24.312457 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:24.312410    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:58:34.312457 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:34.312366    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:58:44.313244 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:44.313200    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 15:58:48.052886 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:48.052858    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:58:51.670167 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:51.670137    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"]
Sep 17 15:58:51.670566 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:51.670391    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container" containerID="cri-o://692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7" gracePeriod=30
Sep 17 15:58:51.670566 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:51.670423    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kube-rbac-proxy" containerID="cri-o://7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87" gracePeriod=30
Sep 17 15:58:52.591026 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:52.590994    2575 generic.go:356] "Generic (PLEG): container finished" podID="320003f2-ac95-45c9-9651-5313908c018b" containerID="7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87" exitCode=2
Sep 17 15:58:52.591195 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:52.591032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerDied","Data":"7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87"}
Sep 17 15:58:54.308068 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:54.308026    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 15:58:55.915908 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:55.915871    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:58:56.046927 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.046899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-cabundle-cert\") pod \"320003f2-ac95-45c9-9651-5313908c018b\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") "
Sep 17 15:58:56.047115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.046940    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/320003f2-ac95-45c9-9651-5313908c018b-proxy-tls\") pod \"320003f2-ac95-45c9-9651-5313908c018b\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") "
Sep 17 15:58:56.047115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.047006    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/320003f2-ac95-45c9-9651-5313908c018b-kserve-provision-location\") pod \"320003f2-ac95-45c9-9651-5313908c018b\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") "
Sep 17 15:58:56.047115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.047022    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-kube-rbac-proxy-sar-config\") pod \"320003f2-ac95-45c9-9651-5313908c018b\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") "
Sep 17 15:58:56.047115 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.047059    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljkjz\" (UniqueName: \"kubernetes.io/projected/320003f2-ac95-45c9-9651-5313908c018b-kube-api-access-ljkjz\") pod \"320003f2-ac95-45c9-9651-5313908c018b\" (UID: \"320003f2-ac95-45c9-9651-5313908c018b\") "
Sep 17 15:58:56.047401 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.047282    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320003f2-ac95-45c9-9651-5313908c018b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "320003f2-ac95-45c9-9651-5313908c018b" (UID: "320003f2-ac95-45c9-9651-5313908c018b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:56.047401 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.047306    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "320003f2-ac95-45c9-9651-5313908c018b" (UID: "320003f2-ac95-45c9-9651-5313908c018b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:56.047484 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.047405    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "320003f2-ac95-45c9-9651-5313908c018b" (UID: "320003f2-ac95-45c9-9651-5313908c018b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:56.049271 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.049248    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320003f2-ac95-45c9-9651-5313908c018b-kube-api-access-ljkjz" (OuterVolumeSpecName: "kube-api-access-ljkjz") pod "320003f2-ac95-45c9-9651-5313908c018b" (UID: "320003f2-ac95-45c9-9651-5313908c018b"). InnerVolumeSpecName "kube-api-access-ljkjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:56.049361 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.049286    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320003f2-ac95-45c9-9651-5313908c018b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "320003f2-ac95-45c9-9651-5313908c018b" (UID: "320003f2-ac95-45c9-9651-5313908c018b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:56.148212 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.148185    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/320003f2-ac95-45c9-9651-5313908c018b-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:56.148212 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.148207    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:56.148212 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.148217    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkjz\" (UniqueName: \"kubernetes.io/projected/320003f2-ac95-45c9-9651-5313908c018b-kube-api-access-ljkjz\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:56.148374 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.148227    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/320003f2-ac95-45c9-9651-5313908c018b-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:56.148374 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.148238    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/320003f2-ac95-45c9-9651-5313908c018b-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 15:58:56.607424 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.607384    2575 generic.go:356] "Generic (PLEG): container finished" podID="320003f2-ac95-45c9-9651-5313908c018b" containerID="692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7" exitCode=0
Sep 17 15:58:56.607616 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.607439    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"
Sep 17 15:58:56.607616 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.607456    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerDied","Data":"692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7"}
Sep 17 15:58:56.607616 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.607511    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb" event={"ID":"320003f2-ac95-45c9-9651-5313908c018b","Type":"ContainerDied","Data":"98f2b35971bee06e169d20cba9308f51dac412a0af4818fc36422423cbc6faab"}
Sep 17 15:58:56.607616 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.607536    2575 scope.go:117] "RemoveContainer" containerID="7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87"
Sep 17 15:58:56.616270 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.616257    2575 scope.go:117] "RemoveContainer" containerID="692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7"
Sep 17 15:58:56.623170 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.623153    2575 scope.go:117] "RemoveContainer" containerID="a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47"
Sep 17 15:58:56.630095 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.630079    2575 scope.go:117] "RemoveContainer" containerID="7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87"
Sep 17 15:58:56.630354 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:58:56.630337    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87\": container with ID starting with 7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87 not found: ID does not exist" containerID="7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87"
Sep 17 15:58:56.630420 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.630365    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87"} err="failed to get container status \"7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87\": rpc error: code = NotFound desc = could not find container \"7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87\": container with ID starting with 7730e6648b1b68578771b76213a1831afa84c6a7e33a7add1bd6b7a260985d87 not found: ID does not exist"
Sep 17 15:58:56.630420 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.630389    2575 scope.go:117] "RemoveContainer" containerID="692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7"
Sep 17 15:58:56.630706 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:58:56.630686    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7\": container with ID starting with 692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7 not found: ID does not exist" containerID="692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7"
Sep 17 15:58:56.630807 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.630710    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7"} err="failed to get container status \"692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7\": rpc error: code = NotFound desc = could not find container \"692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7\": container with ID starting with 692e14876624e5f9244bdb2834a2a3e2711c71e91549c7f32bf7c975f77ed1e7 not found: ID does not exist"
Sep 17 15:58:56.630807 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.630730    2575 scope.go:117] "RemoveContainer" containerID="a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47"
Sep 17 15:58:56.631101 ip-10-0-133-29 kubenswrapper[2575]: E0917 15:58:56.631073    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47\": container with ID starting with a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47 not found: ID does not exist" containerID="a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47"
Sep 17 15:58:56.631194 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.631109    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47"} err="failed to get container status \"a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47\": rpc error: code = NotFound desc = could not find container \"a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47\": container with ID starting with a50ef32f306c4ad60952fbc7a7a95a62d154e5b73ab159f18067ecea4d4b2d47 not found: ID does not exist"
Sep 17 15:58:56.632768 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.632747    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"]
Sep 17 15:58:56.639325 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.639304    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-eef9f-predictor-658d599877-949jb"]
Sep 17 15:58:56.948742 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.948668    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"]
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.948933    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.948944    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.948987    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.948994    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949000    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="storage-initializer"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949006    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="storage-initializer"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949014    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="storage-initializer"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949020    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="storage-initializer"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949026    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949031    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949040    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949045    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949050    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kube-rbac-proxy"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949055    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kube-rbac-proxy"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949109    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kserve-container"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949120    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="agent"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949129    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kube-rbac-proxy"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949140    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="320003f2-ac95-45c9-9651-5313908c018b" containerName="kserve-container"
Sep 17 15:58:56.949210 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.949149    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f90e7d6-0bc4-4590-b5d1-5ada2a3dccb3" containerName="kube-rbac-proxy"
Sep 17 15:58:56.953649 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.953630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:56.956302 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-f5eca-predictor-serving-cert\""
Sep 17 15:58:56.956436 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956304    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:58:56.956436 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956306    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 15:58:56.956436 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956387    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:58:56.956436 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956284    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:58:56.956436 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956433    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-f5eca-kube-rbac-proxy-sar-config\""
Sep 17 15:58:56.956659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.956599    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:58:56.963335 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:56.963312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"]
Sep 17 15:58:57.055021 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.054996    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a18bfd7-18f3-462a-b746-880b422fae5e-proxy-tls\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.055157 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.055036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7nz\" (UniqueName: \"kubernetes.io/projected/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-api-access-8m7nz\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.055157 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.055066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.055157 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.055136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a18bfd7-18f3-462a-b746-880b422fae5e-kserve-provision-location\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.055294 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.055174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-cabundle-cert\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.057527 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.057507    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="320003f2-ac95-45c9-9651-5313908c018b" path="/var/lib/kubelet/pods/320003f2-ac95-45c9-9651-5313908c018b/volumes"
Sep 17 15:58:57.156200 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.156177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7nz\" (UniqueName: \"kubernetes.io/projected/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-api-access-8m7nz\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.156406 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.156213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.156659 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.156633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a18bfd7-18f3-462a-b746-880b422fae5e-kserve-provision-location\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.156787 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.156686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-cabundle-cert\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.156787 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.156752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a18bfd7-18f3-462a-b746-880b422fae5e-proxy-tls\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.157287 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.157258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a18bfd7-18f3-462a-b746-880b422fae5e-kserve-provision-location\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.157418 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.157393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.157553 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.157529    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-cabundle-cert\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.162783 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.162763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a18bfd7-18f3-462a-b746-880b422fae5e-proxy-tls\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.166299 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.166276    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7nz\" (UniqueName: \"kubernetes.io/projected/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-api-access-8m7nz\") pod \"raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.264884 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.264860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:58:57.539892 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.539866    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"]
Sep 17 15:58:57.542583 ip-10-0-133-29 kubenswrapper[2575]: W0917 15:58:57.542545    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a18bfd7_18f3_462a_b746_880b422fae5e.slice/crio-3834ead03dee0e46dca9b4ce937e72db6146d04b7338a61c0590389abce40a0d WatchSource:0}: Error finding container 3834ead03dee0e46dca9b4ce937e72db6146d04b7338a61c0590389abce40a0d: Status 404 returned error can't find the container with id 3834ead03dee0e46dca9b4ce937e72db6146d04b7338a61c0590389abce40a0d
Sep 17 15:58:57.612695 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.612666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerStarted","Data":"ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341"}
Sep 17 15:58:57.612816 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:58:57.612734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerStarted","Data":"3834ead03dee0e46dca9b4ce937e72db6146d04b7338a61c0590389abce40a0d"}
Sep 17 15:59:01.631894 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:01.631860    2575 generic.go:356] "Generic (PLEG): container finished" podID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerID="ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341" exitCode=0
Sep 17 15:59:01.632285 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:01.631935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerDied","Data":"ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341"}
Sep 17 15:59:02.638375 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:02.638347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerStarted","Data":"213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17"}
Sep 17 15:59:02.638725 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:02.638404    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerStarted","Data":"bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f"}
Sep 17 15:59:02.638725 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:02.638592    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:59:02.664748 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:02.664694    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podStartSLOduration=2.855079971 podStartE2EDuration="6.664680792s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809600821s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:02.662785665 +0000 UTC m=+1190.092949447" watchObservedRunningTime="2026-09-17 15:59:02.664680792 +0000 UTC m=+1190.094844574"
Sep 17 15:59:03.642608 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:03.642570    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:59:03.643991 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:03.643936    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:04.646013 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:04.645976    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:09.650384 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:09.650357    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 15:59:09.650858 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:09.650834    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:19.651000 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:19.650944    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:29.651058 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:29.651024    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:39.651486 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:39.651445    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:49.651047 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:49.651006    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 15:59:59.650866 ip-10-0-133-29 kubenswrapper[2575]: I0917 15:59:59.650831    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:00:09.651250 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:09.651165    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:00:18.052852 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.052822    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 16:00:18.727097 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.727068    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"]
Sep 17 16:00:18.730258 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.730242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.734704 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.734680    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:00:18.735194 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.735178    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 16:00:18.747504 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.747485    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"]
Sep 17 16:00:18.872938 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.872900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb04703-a2c6-4775-a2ac-cd65ea195131-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.873149 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.873038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb04703-a2c6-4775-a2ac-cd65ea195131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.873149 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.873074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.873149 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.873103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.873149 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.873142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwkp\" (UniqueName: \"kubernetes.io/projected/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-api-access-vbwkp\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.874322 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.874303    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"]
Sep 17 16:00:18.877544 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.877530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:18.880768 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.880748    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:00:18.904317 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.904290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"]
Sep 17 16:00:18.973599 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb04703-a2c6-4775-a2ac-cd65ea195131-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.973768 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ab566f-a86d-4f1b-b938-49e96c366514-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:18.973768 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973636    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:18.973768 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973653    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhvz\" (UniqueName: \"kubernetes.io/projected/64ab566f-a86d-4f1b-b938-49e96c366514-kube-api-access-pjhvz\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:18.973768 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973741    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ab566f-a86d-4f1b-b938-49e96c366514-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:18.973939 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973781    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:18.973939 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb04703-a2c6-4775-a2ac-cd65ea195131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.973939 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.974154 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.973982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.974154 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.974014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwkp\" (UniqueName: \"kubernetes.io/projected/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-api-access-vbwkp\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.974154 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.974041    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb04703-a2c6-4775-a2ac-cd65ea195131-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.974467 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.974450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.974579 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.974563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.976300 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.976284    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb04703-a2c6-4775-a2ac-cd65ea195131-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:18.984107 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:18.984053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwkp\" (UniqueName: \"kubernetes.io/projected/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-api-access-vbwkp\") pod \"isvc-canary-create-v2-predictor-b59dcb89c-ps5jt\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:19.040546 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.040297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:19.075204 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.075170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.075613 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.075302    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ab566f-a86d-4f1b-b938-49e96c366514-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.075613 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.075350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.075613 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.075374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhvz\" (UniqueName: \"kubernetes.io/projected/64ab566f-a86d-4f1b-b938-49e96c366514-kube-api-access-pjhvz\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.075613 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.075408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ab566f-a86d-4f1b-b938-49e96c366514-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.075883 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.075847    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ab566f-a86d-4f1b-b938-49e96c366514-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.076022 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.076001    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.076081 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.076020    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.078100 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.078078    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ab566f-a86d-4f1b-b938-49e96c366514-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.085765 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.085746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhvz\" (UniqueName: \"kubernetes.io/projected/64ab566f-a86d-4f1b-b938-49e96c366514-kube-api-access-pjhvz\") pod \"isvc-canary-create-predictor-5674b8c456-dhmp5\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.187205 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.187169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:19.320470 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.320441    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"]
Sep 17 16:00:19.323012 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:00:19.322984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb04703_a2c6_4775_a2ac_cd65ea195131.slice/crio-7a1ea1008feb4d47a170c4496f59f6a2b88f1627d4bd79c5b137654c0859c4a9 WatchSource:0}: Error finding container 7a1ea1008feb4d47a170c4496f59f6a2b88f1627d4bd79c5b137654c0859c4a9: Status 404 returned error can't find the container with id 7a1ea1008feb4d47a170c4496f59f6a2b88f1627d4bd79c5b137654c0859c4a9
Sep 17 16:00:19.460286 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.460260    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"]
Sep 17 16:00:19.462411 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:00:19.462388    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ab566f_a86d_4f1b_b938_49e96c366514.slice/crio-b50b41efc628021b7df4e77fdc78ade1ad9ca46dc611bbf8e93e4d91afc5bf23 WatchSource:0}: Error finding container b50b41efc628021b7df4e77fdc78ade1ad9ca46dc611bbf8e93e4d91afc5bf23: Status 404 returned error can't find the container with id b50b41efc628021b7df4e77fdc78ade1ad9ca46dc611bbf8e93e4d91afc5bf23
Sep 17 16:00:19.924291 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.924195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerStarted","Data":"60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890"}
Sep 17 16:00:19.924291 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.924263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerStarted","Data":"b50b41efc628021b7df4e77fdc78ade1ad9ca46dc611bbf8e93e4d91afc5bf23"}
Sep 17 16:00:19.926031 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.926006    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerStarted","Data":"03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6"}
Sep 17 16:00:19.926148 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:19.926057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerStarted","Data":"7a1ea1008feb4d47a170c4496f59f6a2b88f1627d4bd79c5b137654c0859c4a9"}
Sep 17 16:00:23.942325 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:23.942289    2575 generic.go:356] "Generic (PLEG): container finished" podID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerID="03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6" exitCode=0
Sep 17 16:00:23.942821 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:23.942367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerDied","Data":"03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6"}
Sep 17 16:00:23.943928 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:23.943907    2575 generic.go:356] "Generic (PLEG): container finished" podID="64ab566f-a86d-4f1b-b938-49e96c366514" containerID="60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890" exitCode=0
Sep 17 16:00:23.944029 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:23.943978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerDied","Data":"60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890"}
Sep 17 16:00:24.950582 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.950544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerStarted","Data":"484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78"}
Sep 17 16:00:24.951059 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.950658    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerStarted","Data":"5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470"}
Sep 17 16:00:24.951059 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.950972    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:24.952771 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.952747    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerStarted","Data":"ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8"}
Sep 17 16:00:24.952871 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.952794    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerStarted","Data":"b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3"}
Sep 17 16:00:24.953063 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.953047    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:24.953106 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.953087    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:24.954603 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.954578    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:00:24.972992 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.972927    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podStartSLOduration=3.180545086 podStartE2EDuration="6.972914273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792369187s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:24.97170091 +0000 UTC m=+1272.401864690" watchObservedRunningTime="2026-09-17 16:00:24.972914273 +0000 UTC m=+1272.403078054"
Sep 17 16:00:24.992665 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:24.992623    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podStartSLOduration=3.24908876 podStartE2EDuration="6.992613301s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.743524541s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:24.992274815 +0000 UTC m=+1272.422438599" watchObservedRunningTime="2026-09-17 16:00:24.992613301 +0000 UTC m=+1272.422777082"
Sep 17 16:00:25.956550 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:25.956505    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:00:25.957034 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:25.956661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:25.958102 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:25.958079    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:00:26.960432 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:26.960393    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:00:26.992363 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:26.992333    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"]
Sep 17 16:00:26.992595 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:26.992568    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" containerID="cri-o://bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f" gracePeriod=30
Sep 17 16:00:26.992658 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:26.992602    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kube-rbac-proxy" containerID="cri-o://213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17" gracePeriod=30
Sep 17 16:00:27.966429 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:27.966395    2575 generic.go:356] "Generic (PLEG): container finished" podID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerID="213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17" exitCode=2
Sep 17 16:00:27.966779 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:27.966472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerDied","Data":"213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17"}
Sep 17 16:00:28.052111 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:28.052070    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 16:00:29.647232 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:29.647187    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 16:00:30.961074 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:30.961043    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:00:30.961564 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:30.961535    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:00:31.836432 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.836407    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 16:00:31.868512 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868486    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m7nz\" (UniqueName: \"kubernetes.io/projected/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-api-access-8m7nz\") pod \"1a18bfd7-18f3-462a-b746-880b422fae5e\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") "
Sep 17 16:00:31.868654 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868566    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a18bfd7-18f3-462a-b746-880b422fae5e-kserve-provision-location\") pod \"1a18bfd7-18f3-462a-b746-880b422fae5e\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") "
Sep 17 16:00:31.868654 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868591    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-cabundle-cert\") pod \"1a18bfd7-18f3-462a-b746-880b422fae5e\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") "
Sep 17 16:00:31.868654 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868625    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-rbac-proxy-sar-config\") pod \"1a18bfd7-18f3-462a-b746-880b422fae5e\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") "
Sep 17 16:00:31.868654 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868644    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a18bfd7-18f3-462a-b746-880b422fae5e-proxy-tls\") pod \"1a18bfd7-18f3-462a-b746-880b422fae5e\" (UID: \"1a18bfd7-18f3-462a-b746-880b422fae5e\") "
Sep 17 16:00:31.868917 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868886    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a18bfd7-18f3-462a-b746-880b422fae5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a18bfd7-18f3-462a-b746-880b422fae5e" (UID: "1a18bfd7-18f3-462a-b746-880b422fae5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:00:31.869013 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.868991    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a18bfd7-18f3-462a-b746-880b422fae5e" (UID: "1a18bfd7-18f3-462a-b746-880b422fae5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:31.869064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.869016    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a18bfd7-18f3-462a-b746-880b422fae5e" (UID: "1a18bfd7-18f3-462a-b746-880b422fae5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:00:31.870805 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.870777    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a18bfd7-18f3-462a-b746-880b422fae5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a18bfd7-18f3-462a-b746-880b422fae5e" (UID: "1a18bfd7-18f3-462a-b746-880b422fae5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:00:31.870890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.870870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-api-access-8m7nz" (OuterVolumeSpecName: "kube-api-access-8m7nz") pod "1a18bfd7-18f3-462a-b746-880b422fae5e" (UID: "1a18bfd7-18f3-462a-b746-880b422fae5e"). InnerVolumeSpecName "kube-api-access-8m7nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:00:31.964790 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.964711    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:00:31.965235 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.965208    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:00:31.969233 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.969214    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:00:31.969308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.969235    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:00:31.969308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.969245    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a18bfd7-18f3-462a-b746-880b422fae5e-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:00:31.969308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.969271    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7nz\" (UniqueName: \"kubernetes.io/projected/1a18bfd7-18f3-462a-b746-880b422fae5e-kube-api-access-8m7nz\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:00:31.969308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.969278    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a18bfd7-18f3-462a-b746-880b422fae5e-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:00:31.985476 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.985454    2575 generic.go:356] "Generic (PLEG): container finished" podID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerID="bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f" exitCode=0
Sep 17 16:00:31.985577 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.985511    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"
Sep 17 16:00:31.985577 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.985517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerDied","Data":"bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f"}
Sep 17 16:00:31.985650 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.985581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp" event={"ID":"1a18bfd7-18f3-462a-b746-880b422fae5e","Type":"ContainerDied","Data":"3834ead03dee0e46dca9b4ce937e72db6146d04b7338a61c0590389abce40a0d"}
Sep 17 16:00:31.985650 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.985613    2575 scope.go:117] "RemoveContainer" containerID="213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17"
Sep 17 16:00:31.994359 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:31.994339    2575 scope.go:117] "RemoveContainer" containerID="bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f"
Sep 17 16:00:32.001613 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.001596    2575 scope.go:117] "RemoveContainer" containerID="ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341"
Sep 17 16:00:32.008435 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.008420    2575 scope.go:117] "RemoveContainer" containerID="213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17"
Sep 17 16:00:32.008719 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:00:32.008700    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17\": container with ID starting with 213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17 not found: ID does not exist" containerID="213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17"
Sep 17 16:00:32.008765 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.008730    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17"} err="failed to get container status \"213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17\": rpc error: code = NotFound desc = could not find container \"213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17\": container with ID starting with 213810fbe7784474e36f198ca0b6a40bed5cb5388e217234723a01df2c723e17 not found: ID does not exist"
Sep 17 16:00:32.008765 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.008747    2575 scope.go:117] "RemoveContainer" containerID="bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f"
Sep 17 16:00:32.009007 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:00:32.008977    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f\": container with ID starting with bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f not found: ID does not exist" containerID="bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f"
Sep 17 16:00:32.009076 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.009015    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f"} err="failed to get container status \"bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f\": rpc error: code = NotFound desc = could not find container \"bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f\": container with ID starting with bc4dc78f92e459636ae5b2e52d8b9b80ea7a7253f93c079dff87fe335dad210f not found: ID does not exist"
Sep 17 16:00:32.009076 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.009035    2575 scope.go:117] "RemoveContainer" containerID="ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341"
Sep 17 16:00:32.009275 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:00:32.009246    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341\": container with ID starting with ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341 not found: ID does not exist" containerID="ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341"
Sep 17 16:00:32.009358 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.009321    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341"} err="failed to get container status \"ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341\": rpc error: code = NotFound desc = could not find container \"ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341\": container with ID starting with ff4feee81ca3a251da0d448ba16d8945196cf927345e76b81eff94fc6f0a8341 not found: ID does not exist"
Sep 17 16:00:32.011262 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.011241    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"]
Sep 17 16:00:32.016724 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:32.016706    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-f5eca-predictor-6dfbf6967c-n5gnp"]
Sep 17 16:00:33.056436 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:33.056405    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" path="/var/lib/kubelet/pods/1a18bfd7-18f3-462a-b746-880b422fae5e/volumes"
Sep 17 16:00:40.961782 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:40.961740    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:00:41.965861 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:41.965817    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:00:50.961807 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:50.961769    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:00:51.965713 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:00:51.965671    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:01:00.961482 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:00.961438    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:01:01.966044 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:01.966006    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:01:10.961867 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:10.961828    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:01:11.966039 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:11.965996    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:01:20.962160 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:20.962077    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:01:21.965739 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:21.965702    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:01:30.961510 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:30.961475    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 16:01:31.965172 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:31.965136    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:01:40.962347 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:40.962275    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:01:41.966127 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:41.966096    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:01:42.105641 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:42.105613    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"]
Sep 17 16:01:42.105839 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:42.105817    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container" containerID="cri-o://b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3" gracePeriod=30
Sep 17 16:01:42.105910 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:42.105835    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kube-rbac-proxy" containerID="cri-o://ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8" gracePeriod=30
Sep 17 16:01:42.244984 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:42.244937    2575 generic.go:356] "Generic (PLEG): container finished" podID="64ab566f-a86d-4f1b-b938-49e96c366514" containerID="ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8" exitCode=2
Sep 17 16:01:42.245080 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:42.244985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerDied","Data":"ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8"}
Sep 17 16:01:45.957745 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:45.957705    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 16:01:46.244912 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.244886    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:01:46.259470 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.259446    2575 generic.go:356] "Generic (PLEG): container finished" podID="64ab566f-a86d-4f1b-b938-49e96c366514" containerID="b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3" exitCode=0
Sep 17 16:01:46.259547 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.259491    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"
Sep 17 16:01:46.259547 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.259529    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerDied","Data":"b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3"}
Sep 17 16:01:46.259631 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.259591    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5" event={"ID":"64ab566f-a86d-4f1b-b938-49e96c366514","Type":"ContainerDied","Data":"b50b41efc628021b7df4e77fdc78ade1ad9ca46dc611bbf8e93e4d91afc5bf23"}
Sep 17 16:01:46.259685 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.259637    2575 scope.go:117] "RemoveContainer" containerID="ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8"
Sep 17 16:01:46.267205 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.267186    2575 scope.go:117] "RemoveContainer" containerID="b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3"
Sep 17 16:01:46.275146 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.275125    2575 scope.go:117] "RemoveContainer" containerID="60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890"
Sep 17 16:01:46.282394 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.282378    2575 scope.go:117] "RemoveContainer" containerID="ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8"
Sep 17 16:01:46.282650 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:01:46.282631    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8\": container with ID starting with ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8 not found: ID does not exist" containerID="ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8"
Sep 17 16:01:46.282721 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.282656    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8"} err="failed to get container status \"ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8\": rpc error: code = NotFound desc = could not find container \"ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8\": container with ID starting with ff27932b0367402e7992ebd4b65493c1d0e345190dcd0f28a6b12ba31c5babf8 not found: ID does not exist"
Sep 17 16:01:46.282721 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.282670    2575 scope.go:117] "RemoveContainer" containerID="b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3"
Sep 17 16:01:46.282902 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:01:46.282883    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3\": container with ID starting with b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3 not found: ID does not exist" containerID="b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3"
Sep 17 16:01:46.282987 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.282909    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3"} err="failed to get container status \"b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3\": rpc error: code = NotFound desc = could not find container \"b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3\": container with ID starting with b6e6584f8c9ea42bbe2b4606e9953f8fa586cc6fe96ab6943ec8be1a4b19c1b3 not found: ID does not exist"
Sep 17 16:01:46.282987 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.282925    2575 scope.go:117] "RemoveContainer" containerID="60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890"
Sep 17 16:01:46.283186 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:01:46.283169    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890\": container with ID starting with 60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890 not found: ID does not exist" containerID="60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890"
Sep 17 16:01:46.283249 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.283190    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890"} err="failed to get container status \"60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890\": rpc error: code = NotFound desc = could not find container \"60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890\": container with ID starting with 60c9ca47699c1322046ce964e6950252691031b36bf787f696164ca55784d890 not found: ID does not exist"
Sep 17 16:01:46.331545 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.331524    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ab566f-a86d-4f1b-b938-49e96c366514-proxy-tls\") pod \"64ab566f-a86d-4f1b-b938-49e96c366514\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") "
Sep 17 16:01:46.331633 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.331554    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjhvz\" (UniqueName: \"kubernetes.io/projected/64ab566f-a86d-4f1b-b938-49e96c366514-kube-api-access-pjhvz\") pod \"64ab566f-a86d-4f1b-b938-49e96c366514\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") "
Sep 17 16:01:46.331633 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.331582    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ab566f-a86d-4f1b-b938-49e96c366514-kserve-provision-location\") pod \"64ab566f-a86d-4f1b-b938-49e96c366514\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") "
Sep 17 16:01:46.331633 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.331607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-kube-rbac-proxy-sar-config\") pod \"64ab566f-a86d-4f1b-b938-49e96c366514\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") "
Sep 17 16:01:46.331749 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.331703    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-cabundle-cert\") pod \"64ab566f-a86d-4f1b-b938-49e96c366514\" (UID: \"64ab566f-a86d-4f1b-b938-49e96c366514\") "
Sep 17 16:01:46.331975 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.331899    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64ab566f-a86d-4f1b-b938-49e96c366514-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64ab566f-a86d-4f1b-b938-49e96c366514" (UID: "64ab566f-a86d-4f1b-b938-49e96c366514"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:46.332064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.332003    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64ab566f-a86d-4f1b-b938-49e96c366514" (UID: "64ab566f-a86d-4f1b-b938-49e96c366514"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:46.332108 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.332083    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64ab566f-a86d-4f1b-b938-49e96c366514" (UID: "64ab566f-a86d-4f1b-b938-49e96c366514"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:46.333613 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.333593    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ab566f-a86d-4f1b-b938-49e96c366514-kube-api-access-pjhvz" (OuterVolumeSpecName: "kube-api-access-pjhvz") pod "64ab566f-a86d-4f1b-b938-49e96c366514" (UID: "64ab566f-a86d-4f1b-b938-49e96c366514"). InnerVolumeSpecName "kube-api-access-pjhvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:46.333685 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.333671    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ab566f-a86d-4f1b-b938-49e96c366514-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64ab566f-a86d-4f1b-b938-49e96c366514" (UID: "64ab566f-a86d-4f1b-b938-49e96c366514"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:46.432433 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.432409    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64ab566f-a86d-4f1b-b938-49e96c366514-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:46.432433 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.432428    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhvz\" (UniqueName: \"kubernetes.io/projected/64ab566f-a86d-4f1b-b938-49e96c366514-kube-api-access-pjhvz\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:46.432545 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.432437    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64ab566f-a86d-4f1b-b938-49e96c366514-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:46.432545 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.432446    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:46.432545 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.432456    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64ab566f-a86d-4f1b-b938-49e96c366514-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:46.584669 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.584642    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"]
Sep 17 16:01:46.590756 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:46.590736    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-dhmp5"]
Sep 17 16:01:47.056717 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:47.056677    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" path="/var/lib/kubelet/pods/64ab566f-a86d-4f1b-b938-49e96c366514/volumes"
Sep 17 16:01:48.832175 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:48.832145    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"]
Sep 17 16:01:48.832575 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:48.832410    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" containerID="cri-o://5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470" gracePeriod=30
Sep 17 16:01:48.832575 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:48.832411    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kube-rbac-proxy" containerID="cri-o://484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78" gracePeriod=30
Sep 17 16:01:49.273351 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:49.273322    2575 generic.go:356] "Generic (PLEG): container finished" podID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerID="484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78" exitCode=2
Sep 17 16:01:49.273509 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:49.273396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerDied","Data":"484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78"}
Sep 17 16:01:51.961208 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:51.961167    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 16:01:51.965490 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:51.965470    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 16:01:53.077483 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.077457    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:01:53.182924 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.182847    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-cabundle-cert\") pod \"3bb04703-a2c6-4775-a2ac-cd65ea195131\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") "
Sep 17 16:01:53.182924 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.182896    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-rbac-proxy-sar-config\") pod \"3bb04703-a2c6-4775-a2ac-cd65ea195131\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") "
Sep 17 16:01:53.182924 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.182924    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb04703-a2c6-4775-a2ac-cd65ea195131-kserve-provision-location\") pod \"3bb04703-a2c6-4775-a2ac-cd65ea195131\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") "
Sep 17 16:01:53.183215 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.182978    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb04703-a2c6-4775-a2ac-cd65ea195131-proxy-tls\") pod \"3bb04703-a2c6-4775-a2ac-cd65ea195131\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") "
Sep 17 16:01:53.183215 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.182995    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbwkp\" (UniqueName: \"kubernetes.io/projected/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-api-access-vbwkp\") pod \"3bb04703-a2c6-4775-a2ac-cd65ea195131\" (UID: \"3bb04703-a2c6-4775-a2ac-cd65ea195131\") "
Sep 17 16:01:53.183323 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.183264    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb04703-a2c6-4775-a2ac-cd65ea195131-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bb04703-a2c6-4775-a2ac-cd65ea195131" (UID: "3bb04703-a2c6-4775-a2ac-cd65ea195131"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:53.183323 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.183277    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3bb04703-a2c6-4775-a2ac-cd65ea195131" (UID: "3bb04703-a2c6-4775-a2ac-cd65ea195131"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:53.183323 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.183291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3bb04703-a2c6-4775-a2ac-cd65ea195131" (UID: "3bb04703-a2c6-4775-a2ac-cd65ea195131"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:53.185261 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.185238    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb04703-a2c6-4775-a2ac-cd65ea195131-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3bb04703-a2c6-4775-a2ac-cd65ea195131" (UID: "3bb04703-a2c6-4775-a2ac-cd65ea195131"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:53.185365 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.185349    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-api-access-vbwkp" (OuterVolumeSpecName: "kube-api-access-vbwkp") pod "3bb04703-a2c6-4775-a2ac-cd65ea195131" (UID: "3bb04703-a2c6-4775-a2ac-cd65ea195131"). InnerVolumeSpecName "kube-api-access-vbwkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:53.284350 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.284322    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:53.284350 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.284350    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bb04703-a2c6-4775-a2ac-cd65ea195131-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:53.284495 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.284365    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3bb04703-a2c6-4775-a2ac-cd65ea195131-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:53.284495 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.284380    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbwkp\" (UniqueName: \"kubernetes.io/projected/3bb04703-a2c6-4775-a2ac-cd65ea195131-kube-api-access-vbwkp\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:53.284495 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.284392    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3bb04703-a2c6-4775-a2ac-cd65ea195131-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:01:53.288851 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.288826    2575 generic.go:356] "Generic (PLEG): container finished" podID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerID="5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470" exitCode=0
Sep 17 16:01:53.288979 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.288905    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"
Sep 17 16:01:53.288979 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.288911    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerDied","Data":"5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470"}
Sep 17 16:01:53.289084 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.289012    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt" event={"ID":"3bb04703-a2c6-4775-a2ac-cd65ea195131","Type":"ContainerDied","Data":"7a1ea1008feb4d47a170c4496f59f6a2b88f1627d4bd79c5b137654c0859c4a9"}
Sep 17 16:01:53.289084 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.289038    2575 scope.go:117] "RemoveContainer" containerID="484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78"
Sep 17 16:01:53.296929 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.296912    2575 scope.go:117] "RemoveContainer" containerID="5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470"
Sep 17 16:01:53.303905 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.303887    2575 scope.go:117] "RemoveContainer" containerID="03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6"
Sep 17 16:01:53.310464 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.310447    2575 scope.go:117] "RemoveContainer" containerID="484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78"
Sep 17 16:01:53.310724 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:01:53.310702    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78\": container with ID starting with 484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78 not found: ID does not exist" containerID="484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78"
Sep 17 16:01:53.310798 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.310736    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78"} err="failed to get container status \"484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78\": rpc error: code = NotFound desc = could not find container \"484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78\": container with ID starting with 484384ad6132d07ef99ebaf31eec014375a2675f9b2a3f9459b7980745373a78 not found: ID does not exist"
Sep 17 16:01:53.310798 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.310759    2575 scope.go:117] "RemoveContainer" containerID="5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470"
Sep 17 16:01:53.311096 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:01:53.311070    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470\": container with ID starting with 5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470 not found: ID does not exist" containerID="5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470"
Sep 17 16:01:53.311143 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.311100    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470"} err="failed to get container status \"5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470\": rpc error: code = NotFound desc = could not find container \"5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470\": container with ID starting with 5e8b95d9775832873a51806fb5bbbe8c3d05741fad786079ccddb70e55530470 not found: ID does not exist"
Sep 17 16:01:53.311143 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.311115    2575 scope.go:117] "RemoveContainer" containerID="03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6"
Sep 17 16:01:53.311426 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:01:53.311398    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6\": container with ID starting with 03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6 not found: ID does not exist" containerID="03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6"
Sep 17 16:01:53.311519 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.311432    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6"} err="failed to get container status \"03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6\": rpc error: code = NotFound desc = could not find container \"03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6\": container with ID starting with 03af05bb05a7596e6fc7e283f60eadae0d47b8c1e61670d6aa02f03842bcc9e6 not found: ID does not exist"
Sep 17 16:01:53.313266 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.313245    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"]
Sep 17 16:01:53.315815 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:53.315795    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b59dcb89c-ps5jt"]
Sep 17 16:01:54.059427 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059394    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"]
Sep 17 16:01:54.059801 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059783    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kube-rbac-proxy"
Sep 17 16:01:54.059801 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059801    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059813    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059821    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059833    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059842    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059866    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059875    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059886    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="storage-initializer"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059895    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="storage-initializer"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059904    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059913    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059925    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="storage-initializer"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059932    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="storage-initializer"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059941    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059965    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059976    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="storage-initializer"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.059984    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="storage-initializer"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.060054    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.060064    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.060074    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kserve-container"
Sep 17 16:01:54.060078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.060083    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" containerName="kube-rbac-proxy"
Sep 17 16:01:54.060656 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.060091    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ab566f-a86d-4f1b-b938-49e96c366514" containerName="kserve-container"
Sep 17 16:01:54.060656 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.060099    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a18bfd7-18f3-462a-b746-880b422fae5e" containerName="kserve-container"
Sep 17 16:01:54.064935 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.064916    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.067977 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.067938    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:01:54.068218 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.068206    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:01:54.068353 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.068341    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:01:54.068440 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.068422    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 16:01:54.068985 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.068967    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:01:54.069077 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.069010    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:01:54.069077 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.069031    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:01:54.078833 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.078811    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"]
Sep 17 16:01:54.088541 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.088518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.088645 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.088575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.088645 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.088621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fcc5da-4967-43bc-8924-587b8144bf2d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.088645 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.088644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fcc5da-4967-43bc-8924-587b8144bf2d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.088832 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.088710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwlg5\" (UniqueName: \"kubernetes.io/projected/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-api-access-vwlg5\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.147993 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.147965    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"]
Sep 17 16:01:54.151141 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.151127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.154694 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.154678    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:01:54.170729 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.170713    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"]
Sep 17 16:01:54.189791 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.189861 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.189861 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189823    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc23943f-40b4-4238-ad6c-0074a05a713f-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.189861 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.189861 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189857    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.190071 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189875    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fcc5da-4967-43bc-8924-587b8144bf2d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.190071 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189892    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc23943f-40b4-4238-ad6c-0074a05a713f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.190071 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fcc5da-4967-43bc-8924-587b8144bf2d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.190071 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkh2\" (UniqueName: \"kubernetes.io/projected/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-api-access-zjkh2\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.190071 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.189994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwlg5\" (UniqueName: \"kubernetes.io/projected/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-api-access-vwlg5\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.190321 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.190304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fcc5da-4967-43bc-8924-587b8144bf2d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.190437 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.190422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.190498 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.190483    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.192353 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.192337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fcc5da-4967-43bc-8924-587b8144bf2d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.202895 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.202876    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwlg5\" (UniqueName: \"kubernetes.io/projected/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-api-access-vwlg5\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-tn45t\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.290346 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.290326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.290427 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.290353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc23943f-40b4-4238-ad6c-0074a05a713f-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.290427 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.290377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.290427 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.290411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc23943f-40b4-4238-ad6c-0074a05a713f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.290585 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.290442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkh2\" (UniqueName: \"kubernetes.io/projected/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-api-access-zjkh2\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.290853 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.290825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc23943f-40b4-4238-ad6c-0074a05a713f-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.291056 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.291037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.291127 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.291085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.292739 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.292720    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc23943f-40b4-4238-ad6c-0074a05a713f-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.303851 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.303829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkh2\" (UniqueName: \"kubernetes.io/projected/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-api-access-zjkh2\") pod \"isvc-canary-promote-predictor-8499c568cb-wbfhg\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.376299 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.376246    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:01:54.460275 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.460245    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:01:54.652810 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.652787    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"]
Sep 17 16:01:54.655366 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:01:54.655332    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fcc5da_4967_43bc_8924_587b8144bf2d.slice/crio-4ff1ae2469670e7a1c1416df5c942542e87659fb715f6ea801a7b2981d3fab1f WatchSource:0}: Error finding container 4ff1ae2469670e7a1c1416df5c942542e87659fb715f6ea801a7b2981d3fab1f: Status 404 returned error can't find the container with id 4ff1ae2469670e7a1c1416df5c942542e87659fb715f6ea801a7b2981d3fab1f
Sep 17 16:01:54.733989 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:54.733834    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"]
Sep 17 16:01:54.736219 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:01:54.736189    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc23943f_40b4_4238_ad6c_0074a05a713f.slice/crio-5dd85b5804e623b5658f7d6d6b673259cff056ee564c10679c8d41b20ebfc22a WatchSource:0}: Error finding container 5dd85b5804e623b5658f7d6d6b673259cff056ee564c10679c8d41b20ebfc22a: Status 404 returned error can't find the container with id 5dd85b5804e623b5658f7d6d6b673259cff056ee564c10679c8d41b20ebfc22a
Sep 17 16:01:55.057399 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:55.057361    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb04703-a2c6-4775-a2ac-cd65ea195131" path="/var/lib/kubelet/pods/3bb04703-a2c6-4775-a2ac-cd65ea195131/volumes"
Sep 17 16:01:55.300749 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:55.300707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerStarted","Data":"f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496"}
Sep 17 16:01:55.301218 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:55.300784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerStarted","Data":"5dd85b5804e623b5658f7d6d6b673259cff056ee564c10679c8d41b20ebfc22a"}
Sep 17 16:01:55.302468 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:55.302438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerStarted","Data":"7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf"}
Sep 17 16:01:55.302588 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:55.302502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerStarted","Data":"4ff1ae2469670e7a1c1416df5c942542e87659fb715f6ea801a7b2981d3fab1f"}
Sep 17 16:01:59.317605 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:59.317574    2575 generic.go:356] "Generic (PLEG): container finished" podID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerID="7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf" exitCode=0
Sep 17 16:01:59.318082 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:59.317637    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerDied","Data":"7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf"}
Sep 17 16:01:59.319255 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:59.319233    2575 generic.go:356] "Generic (PLEG): container finished" podID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerID="f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496" exitCode=0
Sep 17 16:01:59.319352 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:01:59.319292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerDied","Data":"f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496"}
Sep 17 16:02:00.325376 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.325338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerStarted","Data":"5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e"}
Sep 17 16:02:00.325806 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.325403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerStarted","Data":"84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505"}
Sep 17 16:02:00.325806 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.325615    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:02:00.327745 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.327724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerStarted","Data":"e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72"}
Sep 17 16:02:00.327884 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.327771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerStarted","Data":"1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa"}
Sep 17 16:02:00.328025 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.328004    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:02:00.351917 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.351876    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podStartSLOduration=2.421228621 podStartE2EDuration="6.351863113s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.930634492s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:00.349920643 +0000 UTC m=+1367.780084422" watchObservedRunningTime="2026-09-17 16:02:00.351863113 +0000 UTC m=+1367.782026893"
Sep 17 16:02:00.370590 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:00.370547    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podStartSLOduration=2.369079031 podStartE2EDuration="6.370535543s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.001456512s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:00.369483589 +0000 UTC m=+1367.799647373" watchObservedRunningTime="2026-09-17 16:02:00.370535543 +0000 UTC m=+1367.800699322"
Sep 17 16:02:01.332852 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:01.332822    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:02:01.333277 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:01.332967    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:02:01.333844 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:01.333819    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:01.333924 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:01.333819    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:02.337102 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:02.337050    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:02.337492 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:02.337153    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:07.341442 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:07.341414    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:02:07.341901 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:07.341785    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:02:07.341901 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:07.341803    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:07.342200 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:07.342179    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:17.342312 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:17.342272    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:17.342717 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:17.342272    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:27.342893 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:27.342862    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:27.343302 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:27.342858    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:37.342378 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:37.342345    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:37.342737 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:37.342382    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:47.342097 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:47.342049    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:47.342543 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:47.342217    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:02:57.342591 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:57.342558    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:02:57.342982 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:02:57.342558    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:03:07.342739 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:07.342660    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 16:03:07.343135 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:07.342669    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 16:03:12.052906 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.052868    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:03:12.204872 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.204830    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"]
Sep 17 16:03:12.205100 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.205078    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container" containerID="cri-o://1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa" gracePeriod=30
Sep 17 16:03:12.205215 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.205156    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kube-rbac-proxy" containerID="cri-o://e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72" gracePeriod=30
Sep 17 16:03:12.206017 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.206000    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:03:12.337606 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.337573    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 16:03:12.603721 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.603637    2575 generic.go:356] "Generic (PLEG): container finished" podID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerID="e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72" exitCode=2
Sep 17 16:03:12.603721 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:12.603711    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerDied","Data":"e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72"}
Sep 17 16:03:16.541205 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.541181    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:03:16.608020 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.607925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-rbac-proxy-sar-config\") pod \"fc23943f-40b4-4238-ad6c-0074a05a713f\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") "
Sep 17 16:03:16.608020 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608012    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-cabundle-cert\") pod \"fc23943f-40b4-4238-ad6c-0074a05a713f\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") "
Sep 17 16:03:16.608228 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc23943f-40b4-4238-ad6c-0074a05a713f-kserve-provision-location\") pod \"fc23943f-40b4-4238-ad6c-0074a05a713f\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") "
Sep 17 16:03:16.608228 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608055    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjkh2\" (UniqueName: \"kubernetes.io/projected/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-api-access-zjkh2\") pod \"fc23943f-40b4-4238-ad6c-0074a05a713f\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") "
Sep 17 16:03:16.608228 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc23943f-40b4-4238-ad6c-0074a05a713f-proxy-tls\") pod \"fc23943f-40b4-4238-ad6c-0074a05a713f\" (UID: \"fc23943f-40b4-4238-ad6c-0074a05a713f\") "
Sep 17 16:03:16.608351 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608326    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc23943f-40b4-4238-ad6c-0074a05a713f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc23943f-40b4-4238-ad6c-0074a05a713f" (UID: "fc23943f-40b4-4238-ad6c-0074a05a713f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:03:16.608393 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608334    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc23943f-40b4-4238-ad6c-0074a05a713f" (UID: "fc23943f-40b4-4238-ad6c-0074a05a713f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:16.608393 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.608383    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc23943f-40b4-4238-ad6c-0074a05a713f" (UID: "fc23943f-40b4-4238-ad6c-0074a05a713f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:16.610519 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.610492    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-api-access-zjkh2" (OuterVolumeSpecName: "kube-api-access-zjkh2") pod "fc23943f-40b4-4238-ad6c-0074a05a713f" (UID: "fc23943f-40b4-4238-ad6c-0074a05a713f"). InnerVolumeSpecName "kube-api-access-zjkh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:16.610519 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.610495    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc23943f-40b4-4238-ad6c-0074a05a713f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc23943f-40b4-4238-ad6c-0074a05a713f" (UID: "fc23943f-40b4-4238-ad6c-0074a05a713f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:16.621200 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.621176    2575 generic.go:356] "Generic (PLEG): container finished" podID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerID="1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa" exitCode=0
Sep 17 16:03:16.621308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.621233    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"
Sep 17 16:03:16.621308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.621229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerDied","Data":"1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa"}
Sep 17 16:03:16.621308 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.621299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg" event={"ID":"fc23943f-40b4-4238-ad6c-0074a05a713f","Type":"ContainerDied","Data":"5dd85b5804e623b5658f7d6d6b673259cff056ee564c10679c8d41b20ebfc22a"}
Sep 17 16:03:16.621415 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.621336    2575 scope.go:117] "RemoveContainer" containerID="e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72"
Sep 17 16:03:16.629879 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.629862    2575 scope.go:117] "RemoveContainer" containerID="1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa"
Sep 17 16:03:16.637046 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.637025    2575 scope.go:117] "RemoveContainer" containerID="f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496"
Sep 17 16:03:16.643985 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.643968    2575 scope.go:117] "RemoveContainer" containerID="e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72"
Sep 17 16:03:16.644252 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:03:16.644232    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72\": container with ID starting with e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72 not found: ID does not exist" containerID="e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72"
Sep 17 16:03:16.644410 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.644262    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72"} err="failed to get container status \"e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72\": rpc error: code = NotFound desc = could not find container \"e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72\": container with ID starting with e5554f0cd98433ca2e96ab40e265d7c526ea8d2d99f8586fc35b0373d5cbef72 not found: ID does not exist"
Sep 17 16:03:16.644410 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.644339    2575 scope.go:117] "RemoveContainer" containerID="1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa"
Sep 17 16:03:16.644890 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:03:16.644743    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa\": container with ID starting with 1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa not found: ID does not exist" containerID="1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa"
Sep 17 16:03:16.644890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.644777    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa"} err="failed to get container status \"1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa\": rpc error: code = NotFound desc = could not find container \"1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa\": container with ID starting with 1a8f747ca6bdb76ea31fd9029382d3eb156b21d990644000f71afc319ecf9afa not found: ID does not exist"
Sep 17 16:03:16.644890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.644800    2575 scope.go:117] "RemoveContainer" containerID="f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496"
Sep 17 16:03:16.645210 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:03:16.645152    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496\": container with ID starting with f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496 not found: ID does not exist" containerID="f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496"
Sep 17 16:03:16.645210 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.645174    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496"} err="failed to get container status \"f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496\": rpc error: code = NotFound desc = could not find container \"f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496\": container with ID starting with f2e9bc30000c206520e152d893ae77fddc2b2ac2094229f760a9e93c1758a496 not found: ID does not exist"
Sep 17 16:03:16.646500 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.646480    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"]
Sep 17 16:03:16.651048 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.651027    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-wbfhg"]
Sep 17 16:03:16.709527 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.709494    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc23943f-40b4-4238-ad6c-0074a05a713f-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:03:16.709527 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.709525    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:03:16.709695 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.709540    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc23943f-40b4-4238-ad6c-0074a05a713f-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:03:16.709695 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.709553    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc23943f-40b4-4238-ad6c-0074a05a713f-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:03:16.709695 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:16.709566    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjkh2\" (UniqueName: \"kubernetes.io/projected/fc23943f-40b4-4238-ad6c-0074a05a713f-kube-api-access-zjkh2\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:03:17.056245 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:03:17.056206    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" path="/var/lib/kubelet/pods/fc23943f-40b4-4238-ad6c-0074a05a713f/volumes"
Sep 17 16:04:44.328729 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:44.328643    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"]
Sep 17 16:04:44.329254 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:44.328971    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container" containerID="cri-o://84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505" gracePeriod=30
Sep 17 16:04:44.329254 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:44.329009    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kube-rbac-proxy" containerID="cri-o://5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e" gracePeriod=30
Sep 17 16:04:44.949181 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:44.949146    2575 generic.go:356] "Generic (PLEG): container finished" podID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerID="5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e" exitCode=2
Sep 17 16:04:44.949367 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:44.949214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerDied","Data":"5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e"}
Sep 17 16:04:47.337360 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:47.337331    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 16:04:48.572315 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.572290    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:04:48.738356 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738280    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fcc5da-4967-43bc-8924-587b8144bf2d-kserve-provision-location\") pod \"76fcc5da-4967-43bc-8924-587b8144bf2d\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") "
Sep 17 16:04:48.738356 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738331    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwlg5\" (UniqueName: \"kubernetes.io/projected/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-api-access-vwlg5\") pod \"76fcc5da-4967-43bc-8924-587b8144bf2d\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") "
Sep 17 16:04:48.738555 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738361    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fcc5da-4967-43bc-8924-587b8144bf2d-proxy-tls\") pod \"76fcc5da-4967-43bc-8924-587b8144bf2d\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") "
Sep 17 16:04:48.738555 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738408    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-rbac-proxy-sar-config\") pod \"76fcc5da-4967-43bc-8924-587b8144bf2d\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") "
Sep 17 16:04:48.738555 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738430    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-cabundle-cert\") pod \"76fcc5da-4967-43bc-8924-587b8144bf2d\" (UID: \"76fcc5da-4967-43bc-8924-587b8144bf2d\") "
Sep 17 16:04:48.738723 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738613    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fcc5da-4967-43bc-8924-587b8144bf2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76fcc5da-4967-43bc-8924-587b8144bf2d" (UID: "76fcc5da-4967-43bc-8924-587b8144bf2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:48.738818 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738793    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76fcc5da-4967-43bc-8924-587b8144bf2d" (UID: "76fcc5da-4967-43bc-8924-587b8144bf2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:48.738866 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.738807    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76fcc5da-4967-43bc-8924-587b8144bf2d" (UID: "76fcc5da-4967-43bc-8924-587b8144bf2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:48.740628 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.740603    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fcc5da-4967-43bc-8924-587b8144bf2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76fcc5da-4967-43bc-8924-587b8144bf2d" (UID: "76fcc5da-4967-43bc-8924-587b8144bf2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:48.740628 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.740612    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-api-access-vwlg5" (OuterVolumeSpecName: "kube-api-access-vwlg5") pod "76fcc5da-4967-43bc-8924-587b8144bf2d" (UID: "76fcc5da-4967-43bc-8924-587b8144bf2d"). InnerVolumeSpecName "kube-api-access-vwlg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:48.839779 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.839752    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fcc5da-4967-43bc-8924-587b8144bf2d-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.839779 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.839775    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.839912 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.839784    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fcc5da-4967-43bc-8924-587b8144bf2d-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.839912 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.839793    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fcc5da-4967-43bc-8924-587b8144bf2d-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.839912 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.839801    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwlg5\" (UniqueName: \"kubernetes.io/projected/76fcc5da-4967-43bc-8924-587b8144bf2d-kube-api-access-vwlg5\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:04:48.965214 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.965180    2575 generic.go:356] "Generic (PLEG): container finished" podID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerID="84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505" exitCode=0
Sep 17 16:04:48.965335 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.965240    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"
Sep 17 16:04:48.965335 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.965266    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerDied","Data":"84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505"}
Sep 17 16:04:48.965412 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.965367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t" event={"ID":"76fcc5da-4967-43bc-8924-587b8144bf2d","Type":"ContainerDied","Data":"4ff1ae2469670e7a1c1416df5c942542e87659fb715f6ea801a7b2981d3fab1f"}
Sep 17 16:04:48.965412 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.965408    2575 scope.go:117] "RemoveContainer" containerID="5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e"
Sep 17 16:04:48.973659 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.973646    2575 scope.go:117] "RemoveContainer" containerID="84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505"
Sep 17 16:04:48.980895 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.980882    2575 scope.go:117] "RemoveContainer" containerID="7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf"
Sep 17 16:04:48.987690 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.987629    2575 scope.go:117] "RemoveContainer" containerID="5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e"
Sep 17 16:04:48.988078 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:04:48.987981    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e\": container with ID starting with 5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e not found: ID does not exist" containerID="5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e"
Sep 17 16:04:48.988078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.988017    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e"} err="failed to get container status \"5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e\": rpc error: code = NotFound desc = could not find container \"5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e\": container with ID starting with 5068d6dd2c482bbcea29cd31b241e12608b1e2dffa18c32a0352f371675bdb1e not found: ID does not exist"
Sep 17 16:04:48.988078 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.988039    2575 scope.go:117] "RemoveContainer" containerID="84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505"
Sep 17 16:04:48.988416 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:04:48.988373    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505\": container with ID starting with 84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505 not found: ID does not exist" containerID="84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505"
Sep 17 16:04:48.988416 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.988397    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505"} err="failed to get container status \"84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505\": rpc error: code = NotFound desc = could not find container \"84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505\": container with ID starting with 84807a9bb7cb4f62e40c85b6fe64a5640c85a1124efadd85f8008845d4dc6505 not found: ID does not exist"
Sep 17 16:04:48.988536 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.988419    2575 scope.go:117] "RemoveContainer" containerID="7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf"
Sep 17 16:04:48.988711 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:04:48.988692    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf\": container with ID starting with 7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf not found: ID does not exist" containerID="7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf"
Sep 17 16:04:48.988754 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.988718    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf"} err="failed to get container status \"7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf\": rpc error: code = NotFound desc = could not find container \"7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf\": container with ID starting with 7079b0048079397e8c7edcfb9145bdd5160cbf1e0ebb68d485627fdc24f3ffbf not found: ID does not exist"
Sep 17 16:04:48.989389 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.989374    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"]
Sep 17 16:04:48.995462 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:48.995442    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-tn45t"]
Sep 17 16:04:49.056204 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:49.056184    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" path="/var/lib/kubelet/pods/76fcc5da-4967-43bc-8924-587b8144bf2d/volumes"
Sep 17 16:04:58.664551 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.664514    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"]
Sep 17 16:04:58.664944 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.664929    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="storage-initializer"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.664964    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="storage-initializer"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.664977    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.664985    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.664997    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="storage-initializer"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665006    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="storage-initializer"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665019    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kube-rbac-proxy"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665026    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kube-rbac-proxy"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665037    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kube-rbac-proxy"
Sep 17 16:04:58.665061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kube-rbac-proxy"
Sep 17 16:04:58.665326 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665065    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container"
Sep 17 16:04:58.665326 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665075    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container"
Sep 17 16:04:58.665326 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665148    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kserve-container"
Sep 17 16:04:58.665326 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665159    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kserve-container"
Sep 17 16:04:58.665326 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665168    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc23943f-40b4-4238-ad6c-0074a05a713f" containerName="kube-rbac-proxy"
Sep 17 16:04:58.665326 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.665178    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fcc5da-4967-43bc-8924-587b8144bf2d" containerName="kube-rbac-proxy"
Sep 17 16:04:58.669880 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.669862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.672388 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.672368    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:04:58.672501 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.672369    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 16:04:58.672576 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.672493    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:04:58.673318 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.673302    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 16:04:58.673399 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.673343    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tltb9\""
Sep 17 16:04:58.678292 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.678272    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"]
Sep 17 16:04:58.706350 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.706321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.706495 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.706389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ff4v\" (UniqueName: \"kubernetes.io/projected/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-api-access-2ff4v\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.706495 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.706433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.807572 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.807546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.807752 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.807607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ff4v\" (UniqueName: \"kubernetes.io/projected/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-api-access-2ff4v\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.807752 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.807640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.807752 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:04:58.807736    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 16:04:58.807929 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:04:58.807804    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls podName:ca39fdd7-1e42-42e6-9369-2a27fdef2906 nodeName:}" failed. No retries permitted until 2026-09-17 16:04:59.30778566 +0000 UTC m=+1546.737949423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" (UID: "ca39fdd7-1e42-42e6-9369-2a27fdef2906") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 17 16:04:58.808266 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.808243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:58.820577 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:58.820553    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ff4v\" (UniqueName: \"kubernetes.io/projected/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-api-access-2ff4v\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:59.310470 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:59.310437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:59.312958 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:59.312925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-xbwzc\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:59.580674 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:59.580592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:04:59.861103 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:59.859964    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"]
Sep 17 16:04:59.864779 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:04:59.864748    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca39fdd7_1e42_42e6_9369_2a27fdef2906.slice/crio-1067f5208181fdda1d91c8b165bda99073f8831abe69f02311aa9b6adc834c67 WatchSource:0}: Error finding container 1067f5208181fdda1d91c8b165bda99073f8831abe69f02311aa9b6adc834c67: Status 404 returned error can't find the container with id 1067f5208181fdda1d91c8b165bda99073f8831abe69f02311aa9b6adc834c67
Sep 17 16:04:59.866669 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:04:59.866648    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:05:00.012458 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:00.012423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" event={"ID":"ca39fdd7-1e42-42e6-9369-2a27fdef2906","Type":"ContainerStarted","Data":"1067f5208181fdda1d91c8b165bda99073f8831abe69f02311aa9b6adc834c67"}
Sep 17 16:05:02.021414 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:02.021376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" event={"ID":"ca39fdd7-1e42-42e6-9369-2a27fdef2906","Type":"ContainerStarted","Data":"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"}
Sep 17 16:05:02.021890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:02.021445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" event={"ID":"ca39fdd7-1e42-42e6-9369-2a27fdef2906","Type":"ContainerStarted","Data":"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"}
Sep 17 16:05:02.021890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:02.021581    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:05:02.021890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:02.021722    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:05:02.023105 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:02.023088    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:05:02.041333 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:02.041283    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" podStartSLOduration=2.858345458 podStartE2EDuration="4.041270638s" totalImagesPullingTime="1.18292518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:02.04064362 +0000 UTC m=+1549.470807403" watchObservedRunningTime="2026-09-17 16:05:02.041270638 +0000 UTC m=+1549.471434418"
Sep 17 16:05:09.033266 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:09.033238    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:05:18.658624 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.658587    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"]
Sep 17 16:05:18.663285 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.663264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.665848 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.665824    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:05:18.665982 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.665854    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 16:05:18.665982 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.665867    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:05:18.665982 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.665824    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 16:05:18.676440 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.676414    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"]
Sep 17 16:05:18.747120 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.747094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507ca841-dce0-4113-a263-5d90f26d0324-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.747257 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.747123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.747257 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.747146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.747257 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.747164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/507ca841-dce0-4113-a263-5d90f26d0324-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.747257 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.747246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr86\" (UniqueName: \"kubernetes.io/projected/507ca841-dce0-4113-a263-5d90f26d0324-kube-api-access-qkr86\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.848411 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.848381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr86\" (UniqueName: \"kubernetes.io/projected/507ca841-dce0-4113-a263-5d90f26d0324-kube-api-access-qkr86\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.848521 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.848443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507ca841-dce0-4113-a263-5d90f26d0324-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.848521 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.848471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.848521 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.848499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.848630 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.848523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/507ca841-dce0-4113-a263-5d90f26d0324-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.848904 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.848886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/507ca841-dce0-4113-a263-5d90f26d0324-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.849193 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.849165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.849455 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.849248    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.851100 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.851080    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507ca841-dce0-4113-a263-5d90f26d0324-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.858715 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.858687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr86\" (UniqueName: \"kubernetes.io/projected/507ca841-dce0-4113-a263-5d90f26d0324-kube-api-access-qkr86\") pod \"isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:18.973447 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:18.973360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:19.262654 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:19.262628    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"]
Sep 17 16:05:19.264546 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:05:19.264518    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507ca841_dce0_4113_a263_5d90f26d0324.slice/crio-60f949045ca2adfa63b536d4ec3d157f87ef3f99b958cd51a5dfacd04fdfe96c WatchSource:0}: Error finding container 60f949045ca2adfa63b536d4ec3d157f87ef3f99b958cd51a5dfacd04fdfe96c: Status 404 returned error can't find the container with id 60f949045ca2adfa63b536d4ec3d157f87ef3f99b958cd51a5dfacd04fdfe96c
Sep 17 16:05:20.088288 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:20.088252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerStarted","Data":"b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945"}
Sep 17 16:05:20.088661 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:20.088311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerStarted","Data":"60f949045ca2adfa63b536d4ec3d157f87ef3f99b958cd51a5dfacd04fdfe96c"}
Sep 17 16:05:24.104220 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:24.104182    2575 generic.go:356] "Generic (PLEG): container finished" podID="507ca841-dce0-4113-a263-5d90f26d0324" containerID="b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945" exitCode=0
Sep 17 16:05:24.104592 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:24.104231    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerDied","Data":"b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945"}
Sep 17 16:05:25.110237 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.110200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerStarted","Data":"79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b"}
Sep 17 16:05:25.110647 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.110271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerStarted","Data":"5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7"}
Sep 17 16:05:25.110647 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.110298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerStarted","Data":"ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d"}
Sep 17 16:05:25.110647 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.110586    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:25.110647 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.110636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:25.111934 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.111908    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:05:25.134792 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:25.134756    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podStartSLOduration=3.047431698 podStartE2EDuration="7.134743184s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.087311486s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:25.133210689 +0000 UTC m=+1572.563374468" watchObservedRunningTime="2026-09-17 16:05:25.134743184 +0000 UTC m=+1572.564907030"
Sep 17 16:05:26.113866 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:26.113835    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:26.114354 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:26.113974    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:05:26.115005 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:26.114976    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:05:27.118229 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:27.118187    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:05:27.118601 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:27.118557    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:05:32.122602 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:32.122571    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:05:32.123084 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:32.123038    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:05:32.123395 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:32.123370    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:05:42.123817 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:42.123771    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:05:42.124381 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:42.124349    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:05:52.123471 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:52.123423    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:05:52.124061 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:05:52.123943    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:06:02.123890 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:02.123841    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:06:02.124378 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:02.124320    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:06:12.123593 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:12.123495    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:06:12.124077 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:12.123933    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:06:22.123990 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:22.123929    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:06:22.124487 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:22.124424    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:06:32.123355 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:32.123309    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:06:32.123857 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:32.123791    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:06:42.124329 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:42.124286    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:06:42.124703 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:42.124561    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:06:53.642620 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.642592    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-xbwzc_ca39fdd7-1e42-42e6-9369-2a27fdef2906/kserve-container/0.log"
Sep 17 16:06:53.866099 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.866065    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"]
Sep 17 16:06:53.866932 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.866896    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" containerID="cri-o://ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d" gracePeriod=30
Sep 17 16:06:53.868051 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.868021    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" containerID="cri-o://79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b" gracePeriod=30
Sep 17 16:06:53.868179 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.868158    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" containerID="cri-o://5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7" gracePeriod=30
Sep 17 16:06:53.900012 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.899941    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"]
Sep 17 16:06:53.900217 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.900183    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kserve-container" containerID="cri-o://cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e" gracePeriod=30
Sep 17 16:06:53.900312 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:53.900217    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kube-rbac-proxy" containerID="cri-o://65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab" gracePeriod=30
Sep 17 16:06:54.029370 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.029340    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 16:06:54.148438 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.148415    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:06:54.270504 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.270479    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-rbac-proxy-sar-config\") pod \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") "
Sep 17 16:06:54.270644 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.270525    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls\") pod \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") "
Sep 17 16:06:54.270644 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.270567    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ff4v\" (UniqueName: \"kubernetes.io/projected/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-api-access-2ff4v\") pod \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\" (UID: \"ca39fdd7-1e42-42e6-9369-2a27fdef2906\") "
Sep 17 16:06:54.270794 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.270773    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca39fdd7-1e42-42e6-9369-2a27fdef2906" (UID: "ca39fdd7-1e42-42e6-9369-2a27fdef2906"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:54.272744 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.272717    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-api-access-2ff4v" (OuterVolumeSpecName: "kube-api-access-2ff4v") pod "ca39fdd7-1e42-42e6-9369-2a27fdef2906" (UID: "ca39fdd7-1e42-42e6-9369-2a27fdef2906"). InnerVolumeSpecName "kube-api-access-2ff4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:54.272744 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.272730    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca39fdd7-1e42-42e6-9369-2a27fdef2906" (UID: "ca39fdd7-1e42-42e6-9369-2a27fdef2906"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:06:54.371452 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.371415    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:06:54.371452 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.371452    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca39fdd7-1e42-42e6-9369-2a27fdef2906-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:06:54.371452 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.371463    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ff4v\" (UniqueName: \"kubernetes.io/projected/ca39fdd7-1e42-42e6-9369-2a27fdef2906-kube-api-access-2ff4v\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:06:54.455267 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.455241    2575 generic.go:356] "Generic (PLEG): container finished" podID="507ca841-dce0-4113-a263-5d90f26d0324" containerID="5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7" exitCode=2
Sep 17 16:06:54.455376 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.455313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerDied","Data":"5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7"}
Sep 17 16:06:54.456858 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456832    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerID="65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab" exitCode=2
Sep 17 16:06:54.456858 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456849    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerID="cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e" exitCode=2
Sep 17 16:06:54.457035 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456870    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" event={"ID":"ca39fdd7-1e42-42e6-9369-2a27fdef2906","Type":"ContainerDied","Data":"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"}
Sep 17 16:06:54.457035 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456878    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"
Sep 17 16:06:54.457035 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" event={"ID":"ca39fdd7-1e42-42e6-9369-2a27fdef2906","Type":"ContainerDied","Data":"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"}
Sep 17 16:06:54.457035 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc" event={"ID":"ca39fdd7-1e42-42e6-9369-2a27fdef2906","Type":"ContainerDied","Data":"1067f5208181fdda1d91c8b165bda99073f8831abe69f02311aa9b6adc834c67"}
Sep 17 16:06:54.457035 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.456974    2575 scope.go:117] "RemoveContainer" containerID="65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"
Sep 17 16:06:54.465294 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.465280    2575 scope.go:117] "RemoveContainer" containerID="cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"
Sep 17 16:06:54.472288 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.472271    2575 scope.go:117] "RemoveContainer" containerID="65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"
Sep 17 16:06:54.472572 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:06:54.472553    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab\": container with ID starting with 65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab not found: ID does not exist" containerID="65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"
Sep 17 16:06:54.472628 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.472580    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"} err="failed to get container status \"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab\": rpc error: code = NotFound desc = could not find container \"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab\": container with ID starting with 65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab not found: ID does not exist"
Sep 17 16:06:54.472628 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.472595    2575 scope.go:117] "RemoveContainer" containerID="cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"
Sep 17 16:06:54.472834 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:06:54.472817    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e\": container with ID starting with cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e not found: ID does not exist" containerID="cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"
Sep 17 16:06:54.472871 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.472843    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"} err="failed to get container status \"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e\": rpc error: code = NotFound desc = could not find container \"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e\": container with ID starting with cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e not found: ID does not exist"
Sep 17 16:06:54.472871 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.472859    2575 scope.go:117] "RemoveContainer" containerID="65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"
Sep 17 16:06:54.473092 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.473077    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab"} err="failed to get container status \"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab\": rpc error: code = NotFound desc = could not find container \"65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab\": container with ID starting with 65044da9a03f91eed8741cd32d584b14bd09396831ae76325ba7e688f0e878ab not found: ID does not exist"
Sep 17 16:06:54.473092 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.473092    2575 scope.go:117] "RemoveContainer" containerID="cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"
Sep 17 16:06:54.473290 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.473276    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e"} err="failed to get container status \"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e\": rpc error: code = NotFound desc = could not find container \"cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e\": container with ID starting with cbf4e523bda49c0e31a6e14a15ff516f678d164a78d148c04be2ea4e7153187e not found: ID does not exist"
Sep 17 16:06:54.480632 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.480614    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"]
Sep 17 16:06:54.485036 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:54.485017    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-xbwzc"]
Sep 17 16:06:55.056075 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:55.056047    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" path="/var/lib/kubelet/pods/ca39fdd7-1e42-42e6-9369-2a27fdef2906/volumes"
Sep 17 16:06:57.118725 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:57.118688    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:06:58.475242 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:58.475211    2575 generic.go:356] "Generic (PLEG): container finished" podID="507ca841-dce0-4113-a263-5d90f26d0324" containerID="ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d" exitCode=0
Sep 17 16:06:58.475580 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:06:58.475263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerDied","Data":"ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d"}
Sep 17 16:07:02.119082 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:02.119039    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:07:02.123288 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:02.123260    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:07:02.123644 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:02.123609    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:07.118642 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:07.118603    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:07:07.119105 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:07.118772    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:07:10.468090 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:10.468055    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:10.494476 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:10.494453    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:10.510086 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:10.510053    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:10.526328 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:10.526307    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:11.390171 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:11.390146    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:11.405330 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:11.405306    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:11.418488 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:11.418462    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:11.432117 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:11.432095    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:12.119112 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.119076    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:07:12.123437 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.123409    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:07:12.123775 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.123748    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:12.287872 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.287840    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:12.302658 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.302631    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:12.316882 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.316862    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:12.329833 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:12.329812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:13.163568 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:13.163543    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:13.180745 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:13.180726    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:13.192586 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:13.192557    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:13.205560 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:13.205541    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:14.058355 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:14.058327    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:14.073228 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:14.073206    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:14.087817 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:14.087798    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:14.103819 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:14.103802    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:14.980998 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:14.980976    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:14.996024 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:14.996003    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:15.010389 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:15.010375    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:15.023994 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:15.023966    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:15.911902 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:15.911871    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:15.932846 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:15.932822    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:15.948941 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:15.948921    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:15.964761 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:15.964741    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:16.810749 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:16.810721    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:16.826211 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:16.826190    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:16.841929 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:16.841908    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:16.856772 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:16.856753    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:17.118313 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:17.118232    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:07:17.690463 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:17.690436    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:17.706355 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:17.706331    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:17.723580 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:17.723552    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:17.741838 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:17.741812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:18.636064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:18.636035    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:18.652799 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:18.652773    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:18.667407 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:18.667389    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:18.697862 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:18.697841    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:19.554303 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:19.554273    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:19.570393 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:19.570368    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:19.585712 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:19.585692    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:19.600445 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:19.600424    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:20.509268 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:20.509239    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:20.525545 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:20.525520    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:20.544521 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:20.544500    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:20.558782 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:20.558763    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:21.434674 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:21.434639    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:21.456408 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:21.456390    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:21.473237 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:21.473216    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:21.488782 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:21.488761    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:22.119248 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.119213    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 16:07:22.123580 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.123556    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 16:07:22.123717 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.123689    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:07:22.123929 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.123907    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:22.124036 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.124024    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:07:22.394205 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.394130    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:22.410059 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.410032    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:22.422357 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.422337    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:22.435122 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:22.435106    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:23.328283 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:23.328252    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kserve-container/0.log"
Sep 17 16:07:23.343781 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:23.343757    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/kube-rbac-proxy/0.log"
Sep 17 16:07:23.357009 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:23.356990    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/agent/0.log"
Sep 17 16:07:23.372320 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:23.372301    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp_507ca841-dce0-4113-a263-5d90f26d0324/storage-initializer/0.log"
Sep 17 16:07:24.009729 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.009705    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:07:24.093603 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093579    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/507ca841-dce0-4113-a263-5d90f26d0324-kserve-provision-location\") pod \"507ca841-dce0-4113-a263-5d90f26d0324\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") "
Sep 17 16:07:24.093713 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093616    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-kube-rbac-proxy-sar-config\") pod \"507ca841-dce0-4113-a263-5d90f26d0324\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") "
Sep 17 16:07:24.093713 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093637    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkr86\" (UniqueName: \"kubernetes.io/projected/507ca841-dce0-4113-a263-5d90f26d0324-kube-api-access-qkr86\") pod \"507ca841-dce0-4113-a263-5d90f26d0324\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") "
Sep 17 16:07:24.093713 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093674    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507ca841-dce0-4113-a263-5d90f26d0324-proxy-tls\") pod \"507ca841-dce0-4113-a263-5d90f26d0324\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") "
Sep 17 16:07:24.093849 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093785    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-cabundle-cert\") pod \"507ca841-dce0-4113-a263-5d90f26d0324\" (UID: \"507ca841-dce0-4113-a263-5d90f26d0324\") "
Sep 17 16:07:24.093927 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093897    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507ca841-dce0-4113-a263-5d90f26d0324-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "507ca841-dce0-4113-a263-5d90f26d0324" (UID: "507ca841-dce0-4113-a263-5d90f26d0324"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:24.094023 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.093922    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "507ca841-dce0-4113-a263-5d90f26d0324" (UID: "507ca841-dce0-4113-a263-5d90f26d0324"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:24.094115 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.094097    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/507ca841-dce0-4113-a263-5d90f26d0324-kserve-provision-location\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:07:24.094164 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.094120    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:07:24.094164 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.094143    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "507ca841-dce0-4113-a263-5d90f26d0324" (UID: "507ca841-dce0-4113-a263-5d90f26d0324"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:24.095828 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.095802    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507ca841-dce0-4113-a263-5d90f26d0324-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "507ca841-dce0-4113-a263-5d90f26d0324" (UID: "507ca841-dce0-4113-a263-5d90f26d0324"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:07:24.095919 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.095849    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507ca841-dce0-4113-a263-5d90f26d0324-kube-api-access-qkr86" (OuterVolumeSpecName: "kube-api-access-qkr86") pod "507ca841-dce0-4113-a263-5d90f26d0324" (UID: "507ca841-dce0-4113-a263-5d90f26d0324"). InnerVolumeSpecName "kube-api-access-qkr86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:24.195426 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.195371    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/507ca841-dce0-4113-a263-5d90f26d0324-proxy-tls\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:07:24.195426 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.195393    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/507ca841-dce0-4113-a263-5d90f26d0324-cabundle-cert\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:07:24.195426 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.195401    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkr86\" (UniqueName: \"kubernetes.io/projected/507ca841-dce0-4113-a263-5d90f26d0324-kube-api-access-qkr86\") on node \"ip-10-0-133-29.ec2.internal\" DevicePath \"\""
Sep 17 16:07:24.568708 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.568671    2575 generic.go:356] "Generic (PLEG): container finished" podID="507ca841-dce0-4113-a263-5d90f26d0324" containerID="79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b" exitCode=0
Sep 17 16:07:24.569186 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.568739    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"
Sep 17 16:07:24.569186 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.568764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerDied","Data":"79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b"}
Sep 17 16:07:24.569186 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.568856    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp" event={"ID":"507ca841-dce0-4113-a263-5d90f26d0324","Type":"ContainerDied","Data":"60f949045ca2adfa63b536d4ec3d157f87ef3f99b958cd51a5dfacd04fdfe96c"}
Sep 17 16:07:24.569186 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.568903    2575 scope.go:117] "RemoveContainer" containerID="79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b"
Sep 17 16:07:24.577696 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.577674    2575 scope.go:117] "RemoveContainer" containerID="5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7"
Sep 17 16:07:24.584807 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.584789    2575 scope.go:117] "RemoveContainer" containerID="ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d"
Sep 17 16:07:24.591376 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.591363    2575 scope.go:117] "RemoveContainer" containerID="b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945"
Sep 17 16:07:24.593688 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.593664    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"]
Sep 17 16:07:24.596650 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.596633    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-687f696fd4-9t9bp"]
Sep 17 16:07:24.599213 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.599200    2575 scope.go:117] "RemoveContainer" containerID="79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b"
Sep 17 16:07:24.599489 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:07:24.599472    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b\": container with ID starting with 79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b not found: ID does not exist" containerID="79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b"
Sep 17 16:07:24.599527 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.599496    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b"} err="failed to get container status \"79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b\": rpc error: code = NotFound desc = could not find container \"79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b\": container with ID starting with 79a2764f48a6df331eb3233cecf8f3535e351e927c54300b28fc06b694bb827b not found: ID does not exist"
Sep 17 16:07:24.599527 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.599511    2575 scope.go:117] "RemoveContainer" containerID="5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7"
Sep 17 16:07:24.599741 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:07:24.599726    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7\": container with ID starting with 5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7 not found: ID does not exist" containerID="5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7"
Sep 17 16:07:24.599794 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.599744    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7"} err="failed to get container status \"5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7\": rpc error: code = NotFound desc = could not find container \"5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7\": container with ID starting with 5dcb9ca01050322f0cceb4f244bd52658bf65c4e6399618f264f7e5e339b95c7 not found: ID does not exist"
Sep 17 16:07:24.599794 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.599756    2575 scope.go:117] "RemoveContainer" containerID="ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d"
Sep 17 16:07:24.599985 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:07:24.599972    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d\": container with ID starting with ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d not found: ID does not exist" containerID="ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d"
Sep 17 16:07:24.600039 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.599989    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d"} err="failed to get container status \"ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d\": rpc error: code = NotFound desc = could not find container \"ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d\": container with ID starting with ab7e20a72df8955e914b5c549078f84c55aaa3135c60d9ea3faf7ae792a2249d not found: ID does not exist"
Sep 17 16:07:24.600039 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.600000    2575 scope.go:117] "RemoveContainer" containerID="b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945"
Sep 17 16:07:24.600209 ip-10-0-133-29 kubenswrapper[2575]: E0917 16:07:24.600192    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945\": container with ID starting with b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945 not found: ID does not exist" containerID="b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945"
Sep 17 16:07:24.600264 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:24.600220    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945"} err="failed to get container status \"b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945\": rpc error: code = NotFound desc = could not find container \"b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945\": container with ID starting with b851ccf738b51d660ea74ed0e54b3b74cc8cbeb98f6128c1797ffe50c646d945 not found: ID does not exist"
Sep 17 16:07:25.056153 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:25.056125    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="507ca841-dce0-4113-a263-5d90f26d0324" path="/var/lib/kubelet/pods/507ca841-dce0-4113-a263-5d90f26d0324/volumes"
Sep 17 16:07:26.430853 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.430817    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nxdvm/must-gather-rd6ml"]
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431105    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="storage-initializer"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431120    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="storage-initializer"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431137    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431142    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431149    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431154    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431164    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431169    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431177    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kserve-container"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431181    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kserve-container"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431194    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kube-rbac-proxy"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431201    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kube-rbac-proxy"
Sep 17 16:07:26.431236 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431244    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kube-rbac-proxy"
Sep 17 16:07:26.431659 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431250    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kserve-container"
Sep 17 16:07:26.431659 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431257    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="kube-rbac-proxy"
Sep 17 16:07:26.431659 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431267    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="507ca841-dce0-4113-a263-5d90f26d0324" containerName="agent"
Sep 17 16:07:26.431659 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.431278    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca39fdd7-1e42-42e6-9369-2a27fdef2906" containerName="kserve-container"
Sep 17 16:07:26.435485 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.435462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.438119 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.438099    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxdvm\"/\"openshift-service-ca.crt\""
Sep 17 16:07:26.438208 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.438143    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxdvm\"/\"kube-root-ca.crt\""
Sep 17 16:07:26.439105 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.439088    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nxdvm\"/\"default-dockercfg-mzt7p\""
Sep 17 16:07:26.444168 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.444148    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxdvm/must-gather-rd6ml"]
Sep 17 16:07:26.612737 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.612709    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z679r\" (UniqueName: \"kubernetes.io/projected/3877a7b4-56c5-4327-a0bb-3b13ab11f324-kube-api-access-z679r\") pod \"must-gather-rd6ml\" (UID: \"3877a7b4-56c5-4327-a0bb-3b13ab11f324\") " pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.612888 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.612752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3877a7b4-56c5-4327-a0bb-3b13ab11f324-must-gather-output\") pod \"must-gather-rd6ml\" (UID: \"3877a7b4-56c5-4327-a0bb-3b13ab11f324\") " pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.714082 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.714000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3877a7b4-56c5-4327-a0bb-3b13ab11f324-must-gather-output\") pod \"must-gather-rd6ml\" (UID: \"3877a7b4-56c5-4327-a0bb-3b13ab11f324\") " pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.714210 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.714108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z679r\" (UniqueName: \"kubernetes.io/projected/3877a7b4-56c5-4327-a0bb-3b13ab11f324-kube-api-access-z679r\") pod \"must-gather-rd6ml\" (UID: \"3877a7b4-56c5-4327-a0bb-3b13ab11f324\") " pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.714360 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.714343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3877a7b4-56c5-4327-a0bb-3b13ab11f324-must-gather-output\") pod \"must-gather-rd6ml\" (UID: \"3877a7b4-56c5-4327-a0bb-3b13ab11f324\") " pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.725021 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.724995    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z679r\" (UniqueName: \"kubernetes.io/projected/3877a7b4-56c5-4327-a0bb-3b13ab11f324-kube-api-access-z679r\") pod \"must-gather-rd6ml\" (UID: \"3877a7b4-56c5-4327-a0bb-3b13ab11f324\") " pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:26.743977 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:26.743959    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxdvm/must-gather-rd6ml"
Sep 17 16:07:27.015873 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:27.015851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxdvm/must-gather-rd6ml"]
Sep 17 16:07:27.018273 ip-10-0-133-29 kubenswrapper[2575]: W0917 16:07:27.018248    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3877a7b4_56c5_4327_a0bb_3b13ab11f324.slice/crio-21e6d82b9c8dbf82e8f9fcdeefec8d144397b5cef651f0995cbe818ecd4d7122 WatchSource:0}: Error finding container 21e6d82b9c8dbf82e8f9fcdeefec8d144397b5cef651f0995cbe818ecd4d7122: Status 404 returned error can't find the container with id 21e6d82b9c8dbf82e8f9fcdeefec8d144397b5cef651f0995cbe818ecd4d7122
Sep 17 16:07:27.581637 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:27.581606    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/must-gather-rd6ml" event={"ID":"3877a7b4-56c5-4327-a0bb-3b13ab11f324","Type":"ContainerStarted","Data":"21e6d82b9c8dbf82e8f9fcdeefec8d144397b5cef651f0995cbe818ecd4d7122"}
Sep 17 16:07:28.587739 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:28.587695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/must-gather-rd6ml" event={"ID":"3877a7b4-56c5-4327-a0bb-3b13ab11f324","Type":"ContainerStarted","Data":"09360c7905134814d1c704190a57cf999ae85d0738c8b399ff3a1dbd304a2159"}
Sep 17 16:07:28.588232 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:28.587752    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/must-gather-rd6ml" event={"ID":"3877a7b4-56c5-4327-a0bb-3b13ab11f324","Type":"ContainerStarted","Data":"ffdab73f86695ee8d4fdab58a8463b4e2a93f5e7ed70050efb51f70a0a8ddf61"}
Sep 17 16:07:28.604128 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:28.604064    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nxdvm/must-gather-rd6ml" podStartSLOduration=1.456982047 podStartE2EDuration="2.604045993s" totalImagesPullingTime="1.147063946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:28.603999281 +0000 UTC m=+1696.034163065" watchObservedRunningTime="2026-09-17 16:07:28.604045993 +0000 UTC m=+1696.034209774"
Sep 17 16:07:29.667801 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:29.667769    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4rljc_21f563bc-95bb-4b60-9067-b5252f2950bb/global-pull-secret-syncer/0.log"
Sep 17 16:07:29.874616 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:29.874587    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pwvjs_391bbce6-2f5f-4c3d-9bef-2008106ef630/konnectivity-agent/0.log"
Sep 17 16:07:29.927506 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:29.927431    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-29.ec2.internal_ac07046bb35b604614256dce117e3f6a/haproxy/0.log"
Sep 17 16:07:34.520791 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:34.520698    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rtcld_ef6c41bf-5cd5-4229-bd2e-32d531171207/node-exporter/0.log"
Sep 17 16:07:34.546379 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:34.546353    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rtcld_ef6c41bf-5cd5-4229-bd2e-32d531171207/kube-rbac-proxy/0.log"
Sep 17 16:07:34.576525 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:34.576489    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rtcld_ef6c41bf-5cd5-4229-bd2e-32d531171207/init-textfile/0.log"
Sep 17 16:07:37.519897 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.519857    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"]
Sep 17 16:07:37.524763 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.524737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.534083 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.534059    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"]
Sep 17 16:07:37.610151 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.610112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-lib-modules\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.610339 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.610161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8ts\" (UniqueName: \"kubernetes.io/projected/1396eab6-7daf-4f6e-8352-ee47c004d492-kube-api-access-ff8ts\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.610339 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.610193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-proc\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.610339 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.610267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-sys\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.610339 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.610302    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-podres\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.711592 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8ts\" (UniqueName: \"kubernetes.io/projected/1396eab6-7daf-4f6e-8352-ee47c004d492-kube-api-access-ff8ts\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.711812 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-proc\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.711812 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-sys\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.711812 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-podres\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.711812 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-lib-modules\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.711812 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-sys\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.712157 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-lib-modules\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.712157 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-podres\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.712157 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.711943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1396eab6-7daf-4f6e-8352-ee47c004d492-proc\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.721555 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.721523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8ts\" (UniqueName: \"kubernetes.io/projected/1396eab6-7daf-4f6e-8352-ee47c004d492-kube-api-access-ff8ts\") pod \"perf-node-gather-daemonset-z9x8p\" (UID: \"1396eab6-7daf-4f6e-8352-ee47c004d492\") " pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:37.835742 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:37.835655    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:38.123881 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:38.123796    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"]
Sep 17 16:07:38.638064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:38.637183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p" event={"ID":"1396eab6-7daf-4f6e-8352-ee47c004d492","Type":"ContainerStarted","Data":"4dc4f08decfc3c56466f18c18aa90ef3b102248acadf42cf1fa888f48e6c3885"}
Sep 17 16:07:38.638064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:38.637250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p" event={"ID":"1396eab6-7daf-4f6e-8352-ee47c004d492","Type":"ContainerStarted","Data":"34877b4a22e66e0940cedb06a85b30342feb056ffd046550eb9ab45161476820"}
Sep 17 16:07:38.638064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:38.637988    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:39.260072 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:39.260044    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5rnnw_752d9f57-ac3c-4115-9b3a-ceb4c4fb58de/dns/0.log"
Sep 17 16:07:39.286843 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:39.286814    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5rnnw_752d9f57-ac3c-4115-9b3a-ceb4c4fb58de/kube-rbac-proxy/0.log"
Sep 17 16:07:39.385784 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:39.385746    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7bkx8_9143008c-7b5a-47eb-82ab-ea5001a6d78a/dns-node-resolver/0.log"
Sep 17 16:07:40.019426 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:40.019398    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bl4h7_c28ca2ab-73e6-447b-9eec-6a496d0f801c/node-ca/0.log"
Sep 17 16:07:41.209971 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:41.209931    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gs9v_5682f4af-bd6e-48da-9086-f07848261bdf/serve-healthcheck-canary/0.log"
Sep 17 16:07:41.673460 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:41.673430    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gvzhp_5ed0dd50-7be8-49e8-a57e-14e04452e87f/insights-operator/0.log"
Sep 17 16:07:41.673722 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:41.673707    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gvzhp_5ed0dd50-7be8-49e8-a57e-14e04452e87f/insights-operator/1.log"
Sep 17 16:07:41.697904 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:41.697879    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mrj2w_4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4/kube-rbac-proxy/0.log"
Sep 17 16:07:41.726349 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:41.726326    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mrj2w_4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4/exporter/0.log"
Sep 17 16:07:41.752629 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:41.752610    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mrj2w_4bba0f0a-9d7e-4ee8-9f1a-1bb1d0d018c4/extractor/0.log"
Sep 17 16:07:44.282770 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:44.282745    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-n2jds_4cc6ff97-b737-470f-8e1c-a9b1d82c992c/server/0.log"
Sep 17 16:07:44.433064 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:44.433033    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-vjl29_1e3951de-3730-47ef-bf7a-49b1d22f3571/manager/0.log"
Sep 17 16:07:44.542546 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:44.542481    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-n2l8b_98550bc3-6fd4-403d-96a6-189217ab7a8c/seaweedfs/0.log"
Sep 17 16:07:45.654751 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:45.654727    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p"
Sep 17 16:07:45.672822 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:45.672778    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nxdvm/perf-node-gather-daemonset-z9x8p" podStartSLOduration=8.672762587 podStartE2EDuration="8.672762587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:38.661681795 +0000 UTC m=+1706.091845577" watchObservedRunningTime="2026-09-17 16:07:45.672762587 +0000 UTC m=+1713.102926368"
Sep 17 16:07:49.474970 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:49.474282    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n44s8_7bf6160d-e4bf-4e3c-9884-59ecab0f839d/kube-storage-version-migrator-operator/1.log"
Sep 17 16:07:49.475528 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:49.475325    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n44s8_7bf6160d-e4bf-4e3c-9884-59ecab0f839d/kube-storage-version-migrator-operator/0.log"
Sep 17 16:07:50.828818 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:50.828785    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:07:50.857472 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:50.857442    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/egress-router-binary-copy/0.log"
Sep 17 16:07:50.890799 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:50.890780    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/cni-plugins/0.log"
Sep 17 16:07:50.919007 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:50.918979    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/bond-cni-plugin/0.log"
Sep 17 16:07:50.946478 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:50.946460    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/routeoverride-cni/0.log"
Sep 17 16:07:50.974184 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:50.974166    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/whereabouts-cni-bincopy/0.log"
Sep 17 16:07:51.007013 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:51.006984    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jj5tj_19167536-b708-4f8b-b30e-b96491614264/whereabouts-cni/0.log"
Sep 17 16:07:51.263373 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:51.263343    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lxnls_1408322c-4598-437b-9f34-af740c852bec/kube-multus/0.log"
Sep 17 16:07:51.411415 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:51.411373    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wws4p_2aa01c61-e6a5-47d6-8a03-da7e34a2cee4/network-metrics-daemon/0.log"
Sep 17 16:07:51.436915 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:51.436892    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wws4p_2aa01c61-e6a5-47d6-8a03-da7e34a2cee4/kube-rbac-proxy/0.log"
Sep 17 16:07:52.249623 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.249578    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/ovn-controller/0.log"
Sep 17 16:07:52.272266 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.272225    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/ovn-acl-logging/0.log"
Sep 17 16:07:52.301017 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.300992    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/kube-rbac-proxy-node/0.log"
Sep 17 16:07:52.335503 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.335483    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:07:52.361410 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.361390    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/northd/0.log"
Sep 17 16:07:52.386043 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.386026    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/nbdb/0.log"
Sep 17 16:07:52.412022 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.411943    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/sbdb/0.log"
Sep 17 16:07:52.456089 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:52.456069    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926t8_f7c8321a-6335-4272-b2f3-a1919abbec16/ovnkube-controller/0.log"
Sep 17 16:07:54.047870 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:54.047846    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-s57pz_a9df70ce-5f32-4e2c-88e1-520f5523d090/check-endpoints/0.log"
Sep 17 16:07:54.115763 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:54.115731    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cpn9v_201035b8-6e31-42d8-9bf6-5c1265bdaf76/network-check-target-container/0.log"
Sep 17 16:07:55.288801 ip-10-0-133-29 kubenswrapper[2575]: I0917 16:07:55.288775    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pdk2l_aebc4895-93c5-402a-9073-f9f07b51538e/iptables-alerter/0.log"