Sep 11 16:15:25.489272 ip-10-0-135-5 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 16:15:25.489284 ip-10-0-135-5 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 16:15:25.489294 ip-10-0-135-5 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 16:15:25.489547 ip-10-0-135-5 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 16:15:36.579886 ip-10-0-135-5 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 16:15:36.579905 ip-10-0-135-5 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a3a256b737754b1daf0e752a36b20397 --
Sep 11 16:17:35.265442 ip-10-0-135-5 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:17:35.659400 ip-10-0-135-5 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:35.659400 ip-10-0-135-5 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:17:35.659400 ip-10-0-135-5 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:35.659400 ip-10-0-135-5 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:17:35.659400 ip-10-0-135-5 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:35.667266 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.661577    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:17:35.672133 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672110    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:35.672133 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672131    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:35.672133 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672137    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672140    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672144    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672147    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672150    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672152    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672155    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672158    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672161    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672163    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672166    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672171    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672175    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672178    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672180    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672183    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672186    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672188    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672191    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:35.672236 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672193    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672196    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672198    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672201    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672203    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672206    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672208    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672211    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672214    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672216    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672219    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672223    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672225    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672228    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672231    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672233    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672236    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672239    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672241    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672244    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:35.672886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672247    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672249    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672252    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672255    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672257    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672259    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672263    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672266    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672268    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672271    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672273    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672277    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672279    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672282    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672284    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672287    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672291    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672295    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672298    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:35.673609 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672300    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672303    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672306    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672309    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672311    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672370    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672532    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672555    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672561    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672567    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672573    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672579    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672584    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672589    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672594    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672599    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672604    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672609    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672620    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672624    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:35.674087 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672629    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672633    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672638    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672642    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672654    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672662    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672667    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672672    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672676    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672680    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672690    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672695    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672699    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672705    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672710    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672715    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672721    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672726    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.672731    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674250    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:35.674585 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674265    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674269    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674273    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674277    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674280    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674283    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674285    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674288    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674292    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674294    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674297    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674299    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674302    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674304    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674307    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674310    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674313    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674316    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674318    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674321    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:35.675083 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674323    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674326    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674328    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674331    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674333    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674336    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674339    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674341    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674344    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674346    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674349    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674353    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674356    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674358    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674361    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674363    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674365    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674368    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674370    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674373    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:35.675671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674375    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674378    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674380    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674383    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674385    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674388    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674391    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674394    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674397    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674401    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674405    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674408    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674411    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674414    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674417    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674420    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674422    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674425    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674428    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:35.676239 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674430    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674433    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674435    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674438    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674441    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674445    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674448    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674450    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674454    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674457    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674460    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674462    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674465    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674467    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674470    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674472    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674475    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674477    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674480    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674482    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:35.676708 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674485    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674487    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674497    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674500    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674502    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674506    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674508    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674513    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674516    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674519    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674522    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674524    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674527    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674530    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674533    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674536    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674539    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674544    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674547    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:35.677214 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674945    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674949    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674952    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674955    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674957    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674960    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674963    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674965    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674968    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674970    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674973    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674976    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674978    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674981    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674983    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674987    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674989    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674992    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674994    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674997    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:35.677682 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.674999    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675002    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675004    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675006    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675009    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675011    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675014    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675016    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675019    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675022    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675025    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675028    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675036    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675039    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675057    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675061    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675064    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675066    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675069    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675071    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:35.678192 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675074    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675076    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675079    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675081    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675084    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675087    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675089    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675092    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675096    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675099    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675102    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675104    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675107    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675109    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675111    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675114    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675116    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675119    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675121    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:35.678685 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675124    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675127    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675130    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675133    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675136    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675138    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675141    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675143    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675146    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675150    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675154    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675157    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675161    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675164    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675166    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675169    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675173    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675177    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:35.679169 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675179    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675182    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675184    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675187    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675190    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675193    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675195    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675198    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675201    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675204    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675207    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675209    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675212    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675214    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675216    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675220    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675222    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675225    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675228    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675231    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:35.679601 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675234    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675236    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675321    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675372    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675383    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675390    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675397    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675402    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675408    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675414    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675417    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675420    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675424    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675427    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675430    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675433    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675436    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675440    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675443    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675447    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675451    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675454    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675457    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675460    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:17:35.680101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675463    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675467    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675470    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675474    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675477    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675481    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675484    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675487    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675490    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675492    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675497    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675500    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675502    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675506    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675509    2578 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675511    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675516    2578 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675519    2578 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675522    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675525    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675528    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675532    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675535    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675538    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675541    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:17:35.680671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675544    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675547    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675550    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675553    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675556    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675559    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675561    2578 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675565    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675568    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675571    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675574    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675577    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675580    2578 flags.go:64] FLAG: --help="false"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675583    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675586    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675589    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675592    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675595    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675598    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675601    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675604    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675606    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675609    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:17:35.681285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675612    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675615    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675618    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675621    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675623    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675626    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675629    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675632    2578 flags.go:64] FLAG: --lock-file=""
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675635    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675638    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675641    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675647    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675649    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675652    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675655    2578 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675658    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675662    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675664    2578 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675667    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675672    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675675    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675679    2578 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675683    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675686    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675689    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:17:35.681875 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675692    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675695    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675697    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675700    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675708    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675711    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675714    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675717    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675719    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675725    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675727    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675730    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675733    2578 flags.go:64] FLAG: --port="10250"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675736    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675739    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e2255a86226d9600"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675742    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675745    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675748    2578 flags.go:64] FLAG: --register-node="true"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675751    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675756    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675759    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675762    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675765    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675769    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675772    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:17:35.682581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675775    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675778    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675780    2578 flags.go:64] FLAG: --runonce="false"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675783    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675786    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675789    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675792    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675795    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675798    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675801    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675804    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675807    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675809    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675812    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675815    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675818    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675820    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675823    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675829    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675831    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675837    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675841    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675844    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675847    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675850    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:17:35.683198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675853    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675856    2578 flags.go:64] FLAG: --v="2"
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675864    2578 flags.go:64] FLAG: --version="false"
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675868    2578 flags.go:64] FLAG: --vmodule=""
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675872    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.675875    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.675997    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676001    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676004    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676008    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676011    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676013    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676016    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676019    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676021    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676024    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676027    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676030    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676032    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676035    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676040    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:35.683814 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676061    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676064    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676066    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676069    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676072    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676075    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676084    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676089    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676093    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676098    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676101    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676103    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676106    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676108    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676113    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676115    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676118    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676121    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676123    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676126    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:35.684349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676128    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676131    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676134    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676136    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676139    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676142    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676144    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676147    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676150    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676152    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676155    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676159    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676162    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676165    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676168    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676171    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676173    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676176    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676180    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676182    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:35.684865 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676185    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676187    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676190    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676193    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676195    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676198    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676202    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676207    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676210    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676213    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676216    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676218    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676221    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676223    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676227    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676230    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676234    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676237    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676239    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:35.685372 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676242    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676245    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676248    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676251    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676255    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676257    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676260    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676262    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676265    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676268    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676270    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676274    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676277    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676280    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676282    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676285    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676287    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676290    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676293    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676295    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:35.685851 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676298    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676301    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676303    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676306    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.676309    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.676803    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.684163    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.684186    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.684202    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.684208    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684288    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684295    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684298    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684301    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684304    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:35.686460 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684307    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684310    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684314    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684319    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684322    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684324    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684327    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684330    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684332    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684335    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684338    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684340    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684343    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684345    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684348    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684350    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684354    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684357    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684359    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:35.686840 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684363    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684365    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684368    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684371    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684373    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684378    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684380    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684384    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684386    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684389    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684392    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684405    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684408    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684410    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684415    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684418    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684421    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684425    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684427    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684430    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:35.687335 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684433    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684435    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684438    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684441    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684443    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684446    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684448    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684451    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684453    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684456    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684458    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684461    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684465    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684468    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684471    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684474    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684478    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684481    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684486    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684488    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:35.687830 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684491    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684494    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684496    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684499    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684502    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684504    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684507    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684509    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684512    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684514    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684517    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684519    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684522    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684524    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684527    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684529    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684532    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684534    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684537    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684539    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:35.688349 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684542    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684544    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684546    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684549    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684551    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684554    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684556    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684559    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684561    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684564    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684567    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684570    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684573    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684575    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684577    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:35.688868 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.684583    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684724    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684730    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684733    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684736    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684738    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684741    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684744    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684746    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684749    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684752    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684755    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684757    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684760    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684762    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684765    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684767    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684770    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684772    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684775    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:35.689277 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684777    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684779    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684782    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684784    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684787    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684789    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684792    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684795    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684797    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684801    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684803    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684806    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684809    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684811    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684814    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684817    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684819    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684822    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684824    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:35.689766 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684827    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684829    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684831    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684836    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684839    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684842    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684845    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684848    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684850    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684853    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684856    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684858    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684861    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684863    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684866    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684869    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684872    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684874    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684877    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684880    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:35.690346 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684882    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684886    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684890    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684896    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684899    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684901    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684904    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684906    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684909    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684912    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684914    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684916    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684919    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684921    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684924    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684926    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684930    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684933    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684936    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:35.690835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684939    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684942    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684945    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684948    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684950    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684953    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684955    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684958    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684960    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684963    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684966    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684968    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684971    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684974    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684976    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684979    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684982    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684985    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684987    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684990    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:35.691316 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684992    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:35.684995    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.685000    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.685169    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.687292    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.688185    2578 server.go:1019] "Starting client certificate rotation"
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.688286    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.689128    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:17:35.691835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.690908    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:17:35.697836 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.697804    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:17:35.708619 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.708595    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:17:35.711673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.711656    2578 log.go:25] "Validated CRI v1 image API"
Sep 11 16:17:35.712577 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.712562    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:17:35.719405 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.719383    2578 fs.go:135] Filesystem UUIDs: map[732386ba-ad71-4f93-bb02-3d4c05b9fa2e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b5aef25d-c2e4-4036-988e-f81eb4b97fc1:/dev/nvme0n1p4]
Sep 11 16:17:35.719458 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.719405    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:17:35.725474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.725347    2578 manager.go:216] Machine: {Timestamp:2026-09-11 16:17:35.724007365 +0000 UTC m=+0.341976858 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092724 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27460ba0378be972375fe8f1204e2c SystemUUID:ec27460b-a037-8be9-7237-5fe8f1204e2c BootID:a3a256b7-3775-4b1d-af0e-752a36b20397 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:1d:7a:5e:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:1d:7a:5e:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:50:a4:20:a1:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:17:35.725474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.725471    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:17:35.725579 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.725554    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:17:35.728777 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.728742    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:17:35.728931 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.728779    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-5.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:17:35.730014 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.730001    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:17:35.730066 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.730016    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:17:35.730066 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.730024    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:17:35.730650 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.730638    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:17:35.731508 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.731496    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:17:35.734781 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.734765    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:17:35.734933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.734923    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:17:35.734978 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.734970    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:17:35.735010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.734987    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:17:35.735010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.735002    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:17:35.735098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.735012    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:17:35.735098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.735021    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:17:35.735430 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.735417    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:17:35.735468 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.735440    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:17:35.738244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.738215    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:17:35.740567 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.740552    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:17:35.740631 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.740583    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:17:35.742209 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742194    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:17:35.742244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742226    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:17:35.742244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742240    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:17:35.742312 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742250    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:17:35.742312 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742270    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:17:35.742312 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742282    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:17:35.742312 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742306    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:17:35.742312 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742312    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:17:35.742429 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742320    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:17:35.742429 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742325    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:17:35.742429 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742333    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:17:35.742429 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.742339    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:17:35.743177 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.743165    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:17:35.743219 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.743178    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:17:35.746399 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.746367    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:35.746551 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.746490    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:35.746772 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.746740    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:35.748011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.747993    2578 server.go:1268] "Started kubelet"
Sep 11 16:17:35.748123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.748072    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:17:35.748177 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.748099    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:17:35.748177 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.748172    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:17:35.748953 ip-10-0-135-5 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:17:35.749330 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.749246    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:17:35.749395 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.749379    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:17:35.753222 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.752246    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e641268db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.747959003 +0000 UTC m=+0.365928497,LastTimestamp:2026-09-11 16:17:35.747959003 +0000 UTC m=+0.365928497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.754700 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.754682    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:17:35.755483 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.755468    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:17:35.756223 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.756204    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:17:35.756363 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.756341    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:17:35.756762 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.756745    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:17:35.756762 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.756746    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:17:35.756872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.756773    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:17:35.756872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.756842    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:17:35.756872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.756853    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:17:35.757694 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.757679    2578 factory.go:55] Registering systemd factory
Sep 11 16:17:35.757755 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.757697    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:17:35.757914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.757901    2578 factory.go:153] Registering CRI-O factory
Sep 11 16:17:35.757914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.757914    2578 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:17:35.758011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.757965    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:17:35.758011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.757982    2578 factory.go:103] Registering Raw factory
Sep 11 16:17:35.758011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.758001    2578 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:17:35.758526 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.758510    2578 manager.go:349] Starting recovery of all containers
Sep 11 16:17:35.759227 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.759205    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:17:35.759312 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.759245    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:17:35.759370 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.759336    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:35.771312 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.771285    2578 manager.go:354] Recovery completed
Sep 11 16:17:35.773559 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.773539    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 11 16:17:35.776568 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.776555    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.778515 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.778495    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.778598 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.778529    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.778598 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.778543    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:35.779245 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.779229    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:17:35.779322 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.779246    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:17:35.779322 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.779264    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:17:35.779322 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.779234    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.780022 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.780008    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.780086 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.780034    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.780086 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.780060    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:35.780685 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.780673    2578 policy_none.go:49] "None policy: Start"
Sep 11 16:17:35.780722 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.780689    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:17:35.780722 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.780699    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:17:35.781856 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.781844    2578 policy_none.go:47] "Start"
Sep 11 16:17:35.793673 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.793465    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.795161 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.795101    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.798413 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.798382    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tq2j5"
Sep 11 16:17:35.804511 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.804443    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.812655    2578 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.812690    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.812700    2578 server.go:87] "Starting device plugin registration server"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.813022    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.813032    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.813167    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.813249    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.813260    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.813885    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.813940    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:17:35.829427 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.823780    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.780021566 +0000 UTC m=+0.397991062,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.837813 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.837736    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.780038437 +0000 UTC m=+0.398007930,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.855085 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.854982    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e52620\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.780071271 +0000 UTC m=+0.398040763,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.871343 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.871267    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e68955327  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.823647527 +0000 UTC m=+0.441617020,LastTimestamp:2026-09-11 16:17:35.823647527 +0000 UTC m=+0.441617020,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.888279 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.888245    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:17:35.889614 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.889593    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:17:35.889676 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.889667    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:17:35.889710 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.889697    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:17:35.889816 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.889792    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:17:35.901064 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.901014    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 16:17:35.914174 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.914094    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.915173 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.915150    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.915283 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.915192    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.915283 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.915210    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:35.915283 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.915240    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:35.924465 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.924372    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.915173809 +0000 UTC m=+0.533143303,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.933499 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.933454    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:35.933801 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.933672    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.915198629 +0000 UTC m=+0.533168122,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.942088 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.941974    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e52620\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.915215556 +0000 UTC m=+0.533185052,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:35.969470 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:35.969435    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 11 16:17:35.990670 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.990635    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"]
Sep 11 16:17:35.990750 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.990744    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.992437 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.992419    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.992555 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.992450    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.992555 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.992466    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:35.993824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.993807    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.993969 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.993954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:35.994084 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.993991    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.994602 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.994586    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.994692 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.994589    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.994692 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.994635    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.994692 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.994645    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:35.994692 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.994614    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.994692 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.994684    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:35.996097 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.996082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:35.996179 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.996112    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:35.996963 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.996936    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:35.997040 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.996971    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:35.997040 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:35.996986    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:36.004308 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.004233    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.992434909 +0000 UTC m=+0.610404401,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.011474 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.011399    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.992457912 +0000 UTC m=+0.610427405,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.024352 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.024274    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e52620\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.992472674 +0000 UTC m=+0.610442166,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.031910 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.031886    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.035846 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.035776    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.994605465 +0000 UTC m=+0.612574957,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.036794 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.036777    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.044285 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.044209    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.994622797 +0000 UTC m=+0.612592288,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.058484 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.058457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/243f33b0da41e521566de787222b729f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.ec2.internal\" (UID: \"243f33b0da41e521566de787222b729f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.058631 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.058491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.058631 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.058532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.058631 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.058543    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.994640403 +0000 UTC m=+0.612609895,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.068915 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.068830    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e52620\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.994650432 +0000 UTC m=+0.612619927,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.075969 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.075884    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.994675817 +0000 UTC m=+0.612645310,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.089038 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.088957    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e52620\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.994689004 +0000 UTC m=+0.612658496,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.100936 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.100849    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:35.996954649 +0000 UTC m=+0.614924144,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.108651 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.108569    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:35.996978273 +0000 UTC m=+0.614947765,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.118153 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.118079    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e52620\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e52620  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778547232 +0000 UTC m=+0.396516724,LastTimestamp:2026-09-11 16:17:35.996991922 +0000 UTC m=+0.614961418,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.134219 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.134192    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:36.135207 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.135185    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:36.135309 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.135220    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:36.135309 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.135230    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:36.135309 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.135262    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.152949 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.152920    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.153138 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.153021    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e49ed4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e49ed4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778512596 +0000 UTC m=+0.396482088,LastTimestamp:2026-09-11 16:17:36.13520453 +0000 UTC m=+0.753174021,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.159633 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.159606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.159779 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.159641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.159779 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.159665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/243f33b0da41e521566de787222b729f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.ec2.internal\" (UID: \"243f33b0da41e521566de787222b729f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.159779 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.159719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/243f33b0da41e521566de787222b729f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.ec2.internal\" (UID: \"243f33b0da41e521566de787222b729f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.159779 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.159725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.159779 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.159725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec3255f401d21cf30d3c48f303e873f0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal\" (UID: \"ec3255f401d21cf30d3c48f303e873f0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.163734 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.163660    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.ec2.internal.18d4507e65e4fcd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.ec2.internal,UID:ip-10-0-135-5.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-5.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:35.778536662 +0000 UTC m=+0.396506153,LastTimestamp:2026-09-11 16:17:36.135225532 +0000 UTC m=+0.753195025,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.334527 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.334440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.339819 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.339800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.380808 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.380752    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Sep 11 16:17:36.553253 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.553217    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:36.554668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.554651    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:36.554768 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.554681    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:36.554768 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.554691    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:36.554768 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.554725    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.574563 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.574531    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:36.763238 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.763157    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:36.780720 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.780674    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:36.820975 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.820945    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:36.823600 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.823579    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:36.823720 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.823614    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:36.823720 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.823624    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:36.875296 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.875263    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 16:17:36.972878 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:36.972838    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3255f401d21cf30d3c48f303e873f0.slice/crio-da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0 WatchSource:0}: Error finding container da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0: Status 404 returned error can't find the container with id da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0
Sep 11 16:17:36.973261 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:17:36.973243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243f33b0da41e521566de787222b729f.slice/crio-518c88194e4bf01d7a291ad39b63b7f2404f9e13a9a16bc5809c8345a92a5e0b WatchSource:0}: Error finding container 518c88194e4bf01d7a291ad39b63b7f2404f9e13a9a16bc5809c8345a92a5e0b: Status 404 returned error can't find the container with id 518c88194e4bf01d7a291ad39b63b7f2404f9e13a9a16bc5809c8345a92a5e0b
Sep 11 16:17:36.979553 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:36.979532    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:17:36.987242 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.987162    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507ead7d8369  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:36.979714921 +0000 UTC m=+1.597684414,LastTimestamp:2026-09-11 16:17:36.979714921 +0000 UTC m=+1.597684414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:36.998283 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:36.998203    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-5.ec2.internal.18d4507ead7e89bf  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-5.ec2.internal,UID:243f33b0da41e521566de787222b729f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48526e08a2a06d05806dbcf3ddb069e5182371667d9e9f7d2cf09f60cd247d20\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:36.979782079 +0000 UTC m=+1.597751558,LastTimestamp:2026-09-11 16:17:36.979782079 +0000 UTC m=+1.597751558,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:37.053855 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:37.053820    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:37.183042 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:37.183013    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Sep 11 16:17:37.183209 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:37.183091    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:37.375106 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.374986    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:37.376054 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.376024    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:37.376195 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.376064    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:37.376195 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.376098    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:37.376195 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.376160    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:37.399207 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:37.399171    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:37.754895 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.754806    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:37.895912 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.895843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerStarted","Data":"da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0"}
Sep 11 16:17:37.897566 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:37.897528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal" event={"ID":"243f33b0da41e521566de787222b729f","Type":"ContainerStarted","Data":"518c88194e4bf01d7a291ad39b63b7f2404f9e13a9a16bc5809c8345a92a5e0b"}
Sep 11 16:17:38.768618 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:38.768519    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:38.768618 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:38.768551    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:38.786049 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:38.786012    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Sep 11 16:17:38.858209 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:38.858171    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:38.999344 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:38.999305    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:39.000683 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.000660    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:39.000792 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.000697    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:39.000792 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.000710    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:39.000792 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.000748    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:39.016890 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.016865    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:39.018055 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.018030    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:39.018198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.018068    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:39.018198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.018084    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:39.024185 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.024099    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:39.173188 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.173100    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-5.ec2.internal.18d4507f3019ac8e  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-5.ec2.internal,UID:243f33b0da41e521566de787222b729f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48526e08a2a06d05806dbcf3ddb069e5182371667d9e9f7d2cf09f60cd247d20\" in 2.191s (2.191s including waiting). Image size: 324978947 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.17098715 +0000 UTC m=+3.788956630,LastTimestamp:2026-09-11 16:17:39.17098715 +0000 UTC m=+3.788956630,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:39.183122 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.183019    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f30254ebe  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\" in 2.192s (2.192s including waiting). Image size: 286145739 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.171749566 +0000 UTC m=+3.789719061,LastTimestamp:2026-09-11 16:17:39.171749566 +0000 UTC m=+3.789719061,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:39.257008 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.256792    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-5.ec2.internal.18d4507f348f1f26  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-5.ec2.internal,UID:243f33b0da41e521566de787222b729f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.245793062 +0000 UTC m=+3.863762559,LastTimestamp:2026-09-11 16:17:39.245793062 +0000 UTC m=+3.863762559,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:39.265221 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.265136    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-5.ec2.internal.18d4507f3519a522  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-5.ec2.internal,UID:243f33b0da41e521566de787222b729f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.25487133 +0000 UTC m=+3.872840810,LastTimestamp:2026-09-11 16:17:39.25487133 +0000 UTC m=+3.872840810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:39.424279 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.424192    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f3e51133b  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.409498939 +0000 UTC m=+4.027468439,LastTimestamp:2026-09-11 16:17:39.409498939 +0000 UTC m=+4.027468439,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:39.432352 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.432249    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f3ed8855f  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.418375519 +0000 UTC m=+4.036345004,LastTimestamp:2026-09-11 16:17:39.418375519 +0000 UTC m=+4.036345004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:39.756607 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.756535    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:39.849274 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.849242    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 16:17:39.893077 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.893043    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:39.902835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.902799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal" event={"ID":"243f33b0da41e521566de787222b729f","Type":"ContainerStarted","Data":"146012bcc9d69dae902c16e53f231449c3c76f304c91d3b46d4f0252e2639e04"}
Sep 11 16:17:39.903018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.902815    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:39.904321 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.904298    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:39.904417 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.904334    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:39.904417 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.904344    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:39.904590 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.904565    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:39.904709 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.904659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerStarted","Data":"48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1"}
Sep 11 16:17:39.904753 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.904716    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:39.905682 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.905668    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:39.905763 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.905695    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:39.905763 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:39.905711    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:39.905888 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:39.905875    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:40.757674 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.757646    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:40.908030 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.907997    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:40.908245 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.907997    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:40.909155 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.909112    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:40.909155 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.909151    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:40.909336 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.909167    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:40.909336 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.909121    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:40.909336 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.909228    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:40.909336 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:40.909243    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:40.909534 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:40.909438    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:40.909534 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:40.909509    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:41.757762 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:41.757724    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:41.995653 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:41.995613    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Sep 11 16:17:42.226253 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:42.226200    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:42.227577 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:42.227558    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:42.227651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:42.227595    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:42.227651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:42.227606    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:42.227651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:42.227635    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:42.247429 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:42.247391    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:42.754982 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:42.754959    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:43.305159 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:43.305120    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:43.306550 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:43.306531    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:43.306634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:43.306566    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:43.306634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:43.306577    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:43.569983 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:43.569901    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:43.755595 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:43.755559    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:43.958796 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:43.958764    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:44.758339 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:44.758254    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:45.246663 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:45.246621    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:45.255452 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:45.255420    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 16:17:45.758268 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:45.758229    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:45.814995 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:45.814941    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:17:46.754328 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:46.754290    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:47.760348 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:47.760306    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:48.409596 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:48.409562    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 11 16:17:48.647870 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:48.647826    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:48.649170 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:48.649147    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:48.649311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:48.649185    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:48.649311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:48.649196    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:48.649311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:48.649226    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:48.669028 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:48.668938    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:48.762593 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:48.762560    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:49.757972 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:49.757939    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:50.520581 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:50.520546    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:50.758581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.758548    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:50.928712 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.928686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_ec3255f401d21cf30d3c48f303e873f0/setup/0.log"
Sep 11 16:17:50.928856 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.928720    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3255f401d21cf30d3c48f303e873f0" containerID="48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1" exitCode=1
Sep 11 16:17:50.928856 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.928750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerDied","Data":"48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1"}
Sep 11 16:17:50.928920 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.928876    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:50.929936 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.929919    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:50.930018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.929950    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:50.930018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:50.929967    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:50.930209 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:50.930193    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:50.941566 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:50.941493    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d45081ed2bebb4  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36f416b510fcb7bc202c9983b4402fda0000c1565e3a1c5cab4d877aad9ec460\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:50.93301138 +0000 UTC m=+15.550980859,LastTimestamp:2026-09-11 16:17:50.93301138 +0000 UTC m=+15.550980859,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:51.018847 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:51.018765    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f3e51133b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f3e51133b  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.409498939 +0000 UTC m=+4.027468439,LastTimestamp:2026-09-11 16:17:51.008021782 +0000 UTC m=+15.625991277,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:51.030361 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:51.030283    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f3ed8855f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal.18d4507f3ed8855f  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal,UID:ec3255f401d21cf30d3c48f303e873f0,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.ec2.internal,},FirstTimestamp:2026-09-11 16:17:39.418375519 +0000 UTC m=+4.036345004,LastTimestamp:2026-09-11 16:17:51.015789366 +0000 UTC m=+15.633758858,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.ec2.internal,}"
Sep 11 16:17:51.517290 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.517252    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:51.519495 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.519476    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:51.519564 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.519506    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:51.519564 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.519518    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:51.756627 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.756599    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:51.932193 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.932166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_ec3255f401d21cf30d3c48f303e873f0/setup/0.log"
Sep 11 16:17:51.932370 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.932213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerStarted","Data":"a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a"}
Sep 11 16:17:51.932370 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.932312    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:51.933338 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.933324    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:51.933391 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.933354    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:51.933391 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:51.933364    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:51.933556 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:51.933545    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:52.061401 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:52.061368    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:52.689993 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:52.689961    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 11 16:17:52.756853 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:52.756829    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:53.758463 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:53.758434    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:54.754880 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:54.754852    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:55.419293 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:55.419253    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 11 16:17:55.669284 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:55.669255    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:55.670530 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:55.670486    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:55.670530 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:55.670520    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:55.670530 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:55.670530    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:55.670642 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:55.670556    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:55.689076 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:55.689033    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-5.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:17:55.757122 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:55.757098    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:55.770191 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:55.770157    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:55.815525 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:55.815497    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:17:56.760560 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:56.760530    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:56.940743 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:56.940716    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tq2j5"
Sep 11 16:17:57.688387 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:57.688344    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:17:57.763035 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:57.763009    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:57.778882 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:57.778860    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:57.837652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:57.837630    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:57.944591 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:57.944519    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:12:56 +0000 UTC" deadline="2028-03-15 14:32:54.374560158 +0000 UTC"
Sep 11 16:17:57.944591 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:57.944547    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13222h14m56.43001643s"
Sep 11 16:17:58.097781 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.097756    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.097781 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:58.097783    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.149390 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.149363    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.170202 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.170177    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.237419 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.237336    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.516345 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.516284    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.516345 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:58.516310    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.779490 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.779410    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.797012 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.796988    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:58.857810 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:58.857788    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:59.119385 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:17:59.119363    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:17:59.119385 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:17:59.119386    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:18:01.071584 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:01.071549    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:18:01.089282 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:01.089262    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:18:01.149544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:01.149516    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:18:01.411599 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:01.411576    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:18:01.411681 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:01.411603    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-5.ec2.internal" not found
Sep 11 16:18:02.426107 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.426077    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:18:02.689907 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.689830    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:18:02.690962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.690946    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:18:02.691030 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.690977    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:18:02.691030 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.690986    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:18:02.691030 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.691021    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:18:02.700598 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.700576    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:18:02.700649 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.700603    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-5.ec2.internal\": node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:02.723328 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.723303    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:02.824348 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.824307    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:02.924980 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.924945    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:02.950754 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.950710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_ec3255f401d21cf30d3c48f303e873f0/setup/1.log"
Sep 11 16:18:02.951098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.951085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_ec3255f401d21cf30d3c48f303e873f0/setup/0.log"
Sep 11 16:18:02.951140 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.951116    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3255f401d21cf30d3c48f303e873f0" containerID="a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a" exitCode=1
Sep 11 16:18:02.951172 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.951143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerDied","Data":"a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a"}
Sep 11 16:18:02.951222 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.951212    2578 scope.go:117] "RemoveContainer" containerID="48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1"
Sep 11 16:18:02.951350 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.951338    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:18:02.952224 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.952204    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:18:02.952346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.952235    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:18:02.952346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.952249    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:18:02.952842 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.952436    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.ec2.internal\" not found" node="ip-10-0-135-5.ec2.internal"
Sep 11 16:18:02.952842 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:02.952486    2578 scope.go:117] "RemoveContainer" containerID="48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1"
Sep 11 16:18:02.961961 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.961917    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator_ec3255f401d21cf30d3c48f303e873f0_0 in pod sandbox da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0 from index: no such id: '48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1'" containerID="48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1"
Sep 11 16:18:02.962310 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.961988    2578 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator_ec3255f401d21cf30d3c48f303e873f0_0 in pod sandbox da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0 from index: no such id: '48b29c6301479acaf2ecc1cca2bd9aee40a350fbf22aec3e1251861b4ffb0ad1'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator(ec3255f401d21cf30d3c48f303e873f0)\"" logger="UnhandledError"
Sep 11 16:18:02.962310 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:02.962190    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator(ec3255f401d21cf30d3c48f303e873f0)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" podUID="ec3255f401d21cf30d3c48f303e873f0"
Sep 11 16:18:03.025614 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.025563    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.126269 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.126227    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.227030 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.226953    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.327495 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.327462    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.427962 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.427942    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.528480 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.528419    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.628567 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.628536    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.729182 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.729150    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.829284 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.829212    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.929920 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:03.929897    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:03.953658 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:03.953638    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_ec3255f401d21cf30d3c48f303e873f0/setup/1.log"
Sep 11 16:18:04.030460 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.030428    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.131053 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.131025    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.231708 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.231670    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.332336 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.332311    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.432964 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.432879    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.533397 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.533365    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.633978 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.633949    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.734569 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.734493    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.835631 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.835595    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:04.936367 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:04.936330    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.036957 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.036865    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.137410 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.137381    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.238220 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.238181    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.339321 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.339255    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.439888 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.439849    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.540387 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.540355    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.640988 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.640947    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.742075 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.742022    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.816505 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.816472    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.843152 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.843106    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:05.943898 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:05.943822    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.044483 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.044430    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.145065 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.145008    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.245194 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.245109    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.345679 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.345643    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.446237 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.446204    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.546706 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.546632    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.647253 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.647228    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.747956 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.747918    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.848339 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.848273    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:06.949126 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:06.949097    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.049699 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.049656    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.150421 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.150388    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.250824 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.250800    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.351331 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.351298    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.452264 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.452201    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.553130 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.553101    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.653633 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.653602    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.754518 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.754456    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.855367 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.855332    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:07.956134 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:07.956096    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.056980 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.056925    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.157327 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.157283    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.257559 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.257517    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.357955 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.357936    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.458395 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.458363    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.558929 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.558889    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.659458 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.659375    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.759770 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.759734    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.860270 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.860225    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:08.960351 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:08.960288    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.060799 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.060746    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.161298 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.161258    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.261420 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.261350    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.362041 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.362006    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.462991 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.462961    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.502294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:09.502271    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:18:09.503415 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:09.503385    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:18:09.503514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:09.503431    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:18:09.503514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:09.503475    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:18:09.563129 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.563038    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.663594 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.663560    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.763989 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.763957    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.865031 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.865003    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:09.965937 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:09.965910    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.066449 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.066416    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.166987 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.166928    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.267266 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.267238    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.368202 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.368172    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.468827 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.468759    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.569329 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.569292    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.670287 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.670254    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.771309 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.771236    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.871770 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.871737    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:10.972349 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:10.972325    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.073340 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.073280    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.174225 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.174186    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.274347 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.274311    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.374899 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.374875    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.475891 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.475865    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.576373 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.576339    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-5.ec2.internal\" not found"
Sep 11 16:18:11.625423 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.625368    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:18:11.657420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.657383    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal"
Sep 11 16:18:11.672303 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.672284    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:18:11.673706 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.673692    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal"
Sep 11 16:18:11.683320 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.683299    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:18:11.757923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.757901    2578 apiserver.go:52] "Watching apiserver"
Sep 11 16:18:11.767258 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.767234    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:18:11.767608 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.767587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-k6gfq","kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl","openshift-cluster-node-tuning-operator/tuned-p7mzb","openshift-image-registry/node-ca-c5vjm","openshift-network-diagnostics/network-check-target-8gctq","openshift-network-operator/iptables-alerter-bg5xb","kube-system/global-pull-secret-syncer-p5q6j","openshift-dns/node-resolver-b779c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal","openshift-multus/multus-additional-cni-plugins-86z74","openshift-multus/multus-mvn6l","openshift-multus/network-metrics-daemon-s2ks4","openshift-ovn-kubernetes/ovnkube-node-kqb2x"]
Sep 11 16:18:11.769949 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.769934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.771005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.770979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:11.771109 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.771062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:11.772275 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.772247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.772484 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.772423    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.772857 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.772423    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:18:11.773254 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.773116    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.775626 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.775141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.775626 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.775218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sm5bl\""
Sep 11 16:18:11.775626 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.775349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shf5t\""
Sep 11 16:18:11.775626 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.775595    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:18:11.775626 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.775593    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.775907 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.775845    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.776635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.776602    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:18:11.776733 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.776685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.778039 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.778025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.779033 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.779017    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.779142 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.779091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tbbhq\""
Sep 11 16:18:11.779142 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.779104    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.779324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.779307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.779869 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.779853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.779933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.779915    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.780556 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.780536    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:11.781139 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.781123    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:18:11.781220 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.781160    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vnj9l\""
Sep 11 16:18:11.781822 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.781808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.783132 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.783116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:11.783222 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.783165    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:11.784585 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.784456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.784585 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.784510    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:11.785247 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.785229    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.785362 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.785260    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.785362 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.785270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:18:11.785539 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.785524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:18:11.785607 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.785525    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.786193 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.786176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.786309 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.786294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:18:11.786563 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.786544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xb992\""
Sep 11 16:18:11.786654 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.786636    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator(ec3255f401d21cf30d3c48f303e873f0)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" podUID="ec3255f401d21cf30d3c48f303e873f0"
Sep 11 16:18:11.786693 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.786671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6fgnx\""
Sep 11 16:18:11.786772 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.786756    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kc75s\""
Sep 11 16:18:11.787150 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.787130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:18:11.787231 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.787137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:18:11.787483 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.787456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:18:11.787483 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.787465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.787687 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.787675    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lbjjm\""
Sep 11 16:18:11.790552 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.790533    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ksmmm\""
Sep 11 16:18:11.791834 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.791813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:18:11.791834 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.791828    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.792034 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.791841    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.792034 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.791941    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:18:11.792034 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.791968    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:18:11.792203 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.792069    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:18:11.796441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.796422    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:18:11.816018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.816000    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fnz5"
Sep 11 16:18:11.825877 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.825856    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fnz5"
Sep 11 16:18:11.840617 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.840566    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.ec2.internal" podStartSLOduration=0.840549566 podStartE2EDuration="840.549566ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:11.820032337 +0000 UTC m=+36.438001834" watchObservedRunningTime="2026-09-11 16:18:11.840549566 +0000 UTC m=+36.458519061"
Sep 11 16:18:11.857888 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.857860    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:18:11.889679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f6a39a-b4d2-420b-86d6-6535644a7dce-tmp-dir\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.889679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3da31bc3-f199-4904-9c3d-de3d60ff4815-konnectivity-ca\") pod \"konnectivity-agent-k6gfq\" (UID: \"3da31bc3-f199-4904-9c3d-de3d60ff4815\") " pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:11.889679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-cnibin\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.889679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-host-slash\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-systemd\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-socket-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-socket-dir-parent\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85479\" (UniqueName: \"kubernetes.io/projected/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-kube-api-access-85479\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-lib-modules\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-host\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-tmp\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-cni-bin\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/815ae807-3454-424e-8beb-77f2fdce09bf-multus-daemon-config\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.889903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-device-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-tuned\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-system-cni-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-multus-certs\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.889992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-run-netns\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-ovnkube-config\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02b9217e-d171-4f08-95fa-90bd09c6675a-ovn-node-metrics-cert\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-k8s-cni-cncf-io\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-node-log\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890223    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-var-lib-kubelet\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758m5\" (UniqueName: \"kubernetes.io/projected/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-kube-api-access-758m5\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc1cdc55-6865-430d-8810-8d306526177f-host\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc1cdc55-6865-430d-8810-8d306526177f-serviceca\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-cni-netd\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-env-overrides\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-sys\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/815ae807-3454-424e-8beb-77f2fdce09bf-cni-binary-copy\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-conf-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-slash\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-ovn\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-ovnkube-script-lib\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysconfig\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysctl-d\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-etc-kubernetes\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzqbd\" (UniqueName: \"kubernetes.io/projected/815ae807-3454-424e-8beb-77f2fdce09bf-kube-api-access-wzqbd\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.890729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-kubelet\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-var-lib-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-log-socket\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwfx\" (UniqueName: \"kubernetes.io/projected/02b9217e-d171-4f08-95fa-90bd09c6675a-kube-api-access-xdwfx\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-sys-fs\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-kubernetes\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/179f2ddc-df25-46bf-b640-94826d4f434d-kubelet-config\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-systemd-units\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysctl-conf\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-hostroot\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.890985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfxg\" (UniqueName: \"kubernetes.io/projected/a0f6a39a-b4d2-420b-86d6-6535644a7dce-kube-api-access-8mfxg\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-cni-bin\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbcv\" (UniqueName: \"kubernetes.io/projected/51b154f6-27bb-4554-b269-0cfa1de4bda7-kube-api-access-jkbcv\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:11.891428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0f6a39a-b4d2-420b-86d6-6535644a7dce-hosts-file\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-etc-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n65d\" (UniqueName: \"kubernetes.io/projected/baa82fcf-1fb5-428e-9de2-03e32dec5017-kube-api-access-9n65d\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-systemd\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-kubelet\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-cni-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/179f2ddc-df25-46bf-b640-94826d4f434d-dbus\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-iptables-alerter-script\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3da31bc3-f199-4904-9c3d-de3d60ff4815-agent-certs\") pod \"konnectivity-agent-k6gfq\" (UID: \"3da31bc3-f199-4904-9c3d-de3d60ff4815\") " pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-run\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbfs\" (UniqueName: \"kubernetes.io/projected/fc1cdc55-6865-430d-8810-8d306526177f-kube-api-access-xsbfs\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-netns\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-cni-multus\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrt7\" (UniqueName: \"kubernetes.io/projected/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-kube-api-access-pzrt7\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.891911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-registration-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.892473 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891481    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-modprobe-d\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.892473 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.891494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cnibin\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.912524 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.912497    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:18:11.992035 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbfs\" (UniqueName: \"kubernetes.io/projected/fc1cdc55-6865-430d-8810-8d306526177f-kube-api-access-xsbfs\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.992035 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-netns\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-cni-multus\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrt7\" (UniqueName: \"kubernetes.io/projected/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-kube-api-access-pzrt7\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-registration-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-modprobe-d\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cnibin\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-netns\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-cni-multus\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f6a39a-b4d2-420b-86d6-6535644a7dce-tmp-dir\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-registration-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cnibin\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3da31bc3-f199-4904-9c3d-de3d60ff4815-konnectivity-ca\") pod \"konnectivity-agent-k6gfq\" (UID: \"3da31bc3-f199-4904-9c3d-de3d60ff4815\") " pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-cnibin\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-modprobe-d\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-host-slash\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.992420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-host-slash\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-cnibin\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-systemd\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-socket-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-socket-dir-parent\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85479\" (UniqueName: \"kubernetes.io/projected/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-kube-api-access-85479\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-systemd\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-lib-modules\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-socket-dir-parent\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-host\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-tmp\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-cni-bin\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-host\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-socket-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/815ae807-3454-424e-8beb-77f2fdce09bf-multus-daemon-config\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.993196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992651    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-cni-bin\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-device-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-lib-modules\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-tuned\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-device-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-system-cni-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-multus-certs\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-multus-certs\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3da31bc3-f199-4904-9c3d-de3d60ff4815-konnectivity-ca\") pod \"konnectivity-agent-k6gfq\" (UID: \"3da31bc3-f199-4904-9c3d-de3d60ff4815\") " pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-run-netns\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-ovnkube-config\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-system-cni-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.992979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-run-netns\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.992941    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02b9217e-d171-4f08-95fa-90bd09c6675a-ovn-node-metrics-cert\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.993904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0f6a39a-b4d2-420b-86d6-6535644a7dce-tmp-dir\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.993137    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:12.493121694 +0000 UTC m=+37.111091173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-k8s-cni-cncf-io\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/815ae807-3454-424e-8beb-77f2fdce09bf-multus-daemon-config\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993264    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.993308    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-run-k8s-cni-cncf-io\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-node-log\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.993381    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:12.493365197 +0000 UTC m=+37.111334723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-var-lib-kubelet\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.994652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758m5\" (UniqueName: \"kubernetes.io/projected/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-kube-api-access-758m5\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-node-log\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993500    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-var-lib-kubelet\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc1cdc55-6865-430d-8810-8d306526177f-host\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993533    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc1cdc55-6865-430d-8810-8d306526177f-serviceca\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc1cdc55-6865-430d-8810-8d306526177f-host\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-cni-netd\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-env-overrides\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-cni-netd\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-sys\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/815ae807-3454-424e-8beb-77f2fdce09bf-cni-binary-copy\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-conf-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-slash\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-sys\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-ovn\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.995375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-run-ovn\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-slash\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-conf-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-ovnkube-script-lib\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysconfig\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysctl-d\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-etc-kubernetes\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzqbd\" (UniqueName: \"kubernetes.io/projected/815ae807-3454-424e-8beb-77f2fdce09bf-kube-api-access-wzqbd\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-kubelet\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-var-lib-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-etc-kubernetes\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.993978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-log-socket\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-log-socket\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwfx\" (UniqueName: \"kubernetes.io/projected/02b9217e-d171-4f08-95fa-90bd09c6675a-kube-api-access-xdwfx\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysconfig\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-sys-fs\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.996169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-kubernetes\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc1cdc55-6865-430d-8810-8d306526177f-serviceca\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/179f2ddc-df25-46bf-b640-94826d4f434d-kubelet-config\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/179f2ddc-df25-46bf-b640-94826d4f434d-kubelet-config\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994224    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/815ae807-3454-424e-8beb-77f2fdce09bf-cni-binary-copy\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-var-lib-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-kubelet\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-systemd-units\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysctl-conf\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-hostroot\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-kubernetes\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfxg\" (UniqueName: \"kubernetes.io/projected/a0f6a39a-b4d2-420b-86d6-6535644a7dce-kube-api-access-8mfxg\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-cni-bin\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-systemd-units\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbcv\" (UniqueName: \"kubernetes.io/projected/51b154f6-27bb-4554-b269-0cfa1de4bda7-kube-api-access-jkbcv\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:11.997005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0f6a39a-b4d2-420b-86d6-6535644a7dce-hosts-file\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-etc-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n65d\" (UniqueName: \"kubernetes.io/projected/baa82fcf-1fb5-428e-9de2-03e32dec5017-kube-api-access-9n65d\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-systemd\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysctl-d\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-kubelet\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.994970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-cni-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/179f2ddc-df25-46bf-b640-94826d4f434d-dbus\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-iptables-alerter-script\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3da31bc3-f199-4904-9c3d-de3d60ff4815-agent-certs\") pod \"konnectivity-agent-k6gfq\" (UID: \"3da31bc3-f199-4904-9c3d-de3d60ff4815\") " pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-run\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-run\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-host-var-lib-kubelet\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baa82fcf-1fb5-428e-9de2-03e32dec5017-sys-fs\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:11.999988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-env-overrides\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-multus-cni-dir\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-sysctl-conf\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.995800    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/179f2ddc-df25-46bf-b640-94826d4f434d-dbus\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-host-cni-bin\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0f6a39a-b4d2-420b-86d6-6535644a7dce-hosts-file\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02b9217e-d171-4f08-95fa-90bd09c6675a-etc-openvswitch\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/815ae807-3454-424e-8beb-77f2fdce09bf-hostroot\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-systemd\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-iptables-alerter-script\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.996880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-ovnkube-script-lib\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.997782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02b9217e-d171-4f08-95fa-90bd09c6675a-ovnkube-config\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.997902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-tmp\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.998424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-etc-tuned\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:11.998528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3da31bc3-f199-4904-9c3d-de3d60ff4815-agent-certs\") pod \"konnectivity-agent-k6gfq\" (UID: \"3da31bc3-f199-4904-9c3d-de3d60ff4815\") " pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.999702    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.999729    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:12.000544 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.999745    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:12.001327 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:11.999817    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:12.499797931 +0000 UTC m=+37.117767433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:12.001916 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.001626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02b9217e-d171-4f08-95fa-90bd09c6675a-ovn-node-metrics-cert\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.002684 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.002657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbfs\" (UniqueName: \"kubernetes.io/projected/fc1cdc55-6865-430d-8810-8d306526177f-kube-api-access-xsbfs\") pod \"node-ca-c5vjm\" (UID: \"fc1cdc55-6865-430d-8810-8d306526177f\") " pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:12.004323 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.004091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-758m5\" (UniqueName: \"kubernetes.io/projected/f0e6493f-bb8e-4c87-8f2e-9bb5a6861912-kube-api-access-758m5\") pod \"tuned-p7mzb\" (UID: \"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912\") " pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:12.004428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.004392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrt7\" (UniqueName: \"kubernetes.io/projected/39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6-kube-api-access-pzrt7\") pod \"iptables-alerter-bg5xb\" (UID: \"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6\") " pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:12.004760 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.004735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzqbd\" (UniqueName: \"kubernetes.io/projected/815ae807-3454-424e-8beb-77f2fdce09bf-kube-api-access-wzqbd\") pod \"multus-mvn6l\" (UID: \"815ae807-3454-424e-8beb-77f2fdce09bf\") " pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:12.005012 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.004994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85479\" (UniqueName: \"kubernetes.io/projected/a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0-kube-api-access-85479\") pod \"multus-additional-cni-plugins-86z74\" (UID: \"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0\") " pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:12.005181 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.005159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfxg\" (UniqueName: \"kubernetes.io/projected/a0f6a39a-b4d2-420b-86d6-6535644a7dce-kube-api-access-8mfxg\") pod \"node-resolver-b779c\" (UID: \"a0f6a39a-b4d2-420b-86d6-6535644a7dce\") " pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:12.005365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.005343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwfx\" (UniqueName: \"kubernetes.io/projected/02b9217e-d171-4f08-95fa-90bd09c6675a-kube-api-access-xdwfx\") pod \"ovnkube-node-kqb2x\" (UID: \"02b9217e-d171-4f08-95fa-90bd09c6675a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.005672 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.005655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n65d\" (UniqueName: \"kubernetes.io/projected/baa82fcf-1fb5-428e-9de2-03e32dec5017-kube-api-access-9n65d\") pod \"aws-ebs-csi-driver-node-rl8gl\" (UID: \"baa82fcf-1fb5-428e-9de2-03e32dec5017\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:12.006605 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.006577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbcv\" (UniqueName: \"kubernetes.io/projected/51b154f6-27bb-4554-b269-0cfa1de4bda7-kube-api-access-jkbcv\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:12.081290 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.081240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bg5xb"
Sep 11 16:18:12.087824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.087795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl"
Sep 11 16:18:12.091029 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.091006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39aa53cb_d4d0_422a_9f4e_abc9b2d8e4a6.slice/crio-b75205d7a1d47846bf830c70095e9402af3786c5a64572274d72f76309805320 WatchSource:0}: Error finding container b75205d7a1d47846bf830c70095e9402af3786c5a64572274d72f76309805320: Status 404 returned error can't find the container with id b75205d7a1d47846bf830c70095e9402af3786c5a64572274d72f76309805320
Sep 11 16:18:12.094427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.094406    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86z74"
Sep 11 16:18:12.099741 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.099723    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p7mzb"
Sep 11 16:18:12.104883 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.104861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c5vjm"
Sep 11 16:18:12.108987 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.108966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mvn6l"
Sep 11 16:18:12.109647 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.109621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e6493f_bb8e_4c87_8f2e_9bb5a6861912.slice/crio-7ea71b9bf088d5c4f7df7020e147b85e0488b7d28613449261596cb4f2d5949d WatchSource:0}: Error finding container 7ea71b9bf088d5c4f7df7020e147b85e0488b7d28613449261596cb4f2d5949d: Status 404 returned error can't find the container with id 7ea71b9bf088d5c4f7df7020e147b85e0488b7d28613449261596cb4f2d5949d
Sep 11 16:18:12.110373 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.110353    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0aef5d9_403b_4e8e_b0c0_5b27b5b89bd0.slice/crio-407ba99b7d83d18165024ecd6eb5d39f734429c8789362af1d049e2f363abe0e WatchSource:0}: Error finding container 407ba99b7d83d18165024ecd6eb5d39f734429c8789362af1d049e2f363abe0e: Status 404 returned error can't find the container with id 407ba99b7d83d18165024ecd6eb5d39f734429c8789362af1d049e2f363abe0e
Sep 11 16:18:12.113674 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.113651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b779c"
Sep 11 16:18:12.115317 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.115295    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1cdc55_6865_430d_8810_8d306526177f.slice/crio-d9a8367009539ed704ac7c38d69f829cb4a6b68432c596f68314f185a0ca49f3 WatchSource:0}: Error finding container d9a8367009539ed704ac7c38d69f829cb4a6b68432c596f68314f185a0ca49f3: Status 404 returned error can't find the container with id d9a8367009539ed704ac7c38d69f829cb4a6b68432c596f68314f185a0ca49f3
Sep 11 16:18:12.117577 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.117556    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815ae807_3454_424e_8beb_77f2fdce09bf.slice/crio-88d05acf5a91912670582d142c156d6eecd67d3f1b3024ef2af6a2cfbd843b95 WatchSource:0}: Error finding container 88d05acf5a91912670582d142c156d6eecd67d3f1b3024ef2af6a2cfbd843b95: Status 404 returned error can't find the container with id 88d05acf5a91912670582d142c156d6eecd67d3f1b3024ef2af6a2cfbd843b95
Sep 11 16:18:12.118196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.118180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:12.122654 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.122612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:12.122948 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.122913    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f6a39a_b4d2_420b_86d6_6535644a7dce.slice/crio-80cf0ede3fab888e14b2c7c753e627c458b4ef58b187d15224562bba11a2e956 WatchSource:0}: Error finding container 80cf0ede3fab888e14b2c7c753e627c458b4ef58b187d15224562bba11a2e956: Status 404 returned error can't find the container with id 80cf0ede3fab888e14b2c7c753e627c458b4ef58b187d15224562bba11a2e956
Sep 11 16:18:12.127145 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.127123    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da31bc3_f199_4904_9c3d_de3d60ff4815.slice/crio-88c23d657333934fc328ea90110bb2faa4a915ea2d3a767339186b719f8868d8 WatchSource:0}: Error finding container 88c23d657333934fc328ea90110bb2faa4a915ea2d3a767339186b719f8868d8: Status 404 returned error can't find the container with id 88c23d657333934fc328ea90110bb2faa4a915ea2d3a767339186b719f8868d8
Sep 11 16:18:12.133406 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:12.133386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b9217e_d171_4f08_95fa_90bd09c6675a.slice/crio-6837c9bb02085f17e0dde4c6393d3f99801c75b4592bd7efc37f6d70800cb909 WatchSource:0}: Error finding container 6837c9bb02085f17e0dde4c6393d3f99801c75b4592bd7efc37f6d70800cb909: Status 404 returned error can't find the container with id 6837c9bb02085f17e0dde4c6393d3f99801c75b4592bd7efc37f6d70800cb909
Sep 11 16:18:12.500126 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.500085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:12.500672 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.500135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:12.500672 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.500189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:12.500672 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500334    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:12.500672 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500355    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:12.500672 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500368    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:12.500672 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500436    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:13.500416908 +0000 UTC m=+38.118386410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:12.500955 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500776    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:12.500955 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500822    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:13.500809377 +0000 UTC m=+38.118778859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:12.500955 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500861    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:12.500955 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:12.500895    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:13.500877937 +0000 UTC m=+38.118847416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:12.688281 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.687945    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:18:12.829365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.829201    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:13:11 +0000 UTC" deadline="2028-02-13 04:47:02.735428595 +0000 UTC"
Sep 11 16:18:12.829365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.829241    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12468h28m49.906193064s"
Sep 11 16:18:12.969937 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.969801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c5vjm" event={"ID":"fc1cdc55-6865-430d-8810-8d306526177f","Type":"ContainerStarted","Data":"d9a8367009539ed704ac7c38d69f829cb4a6b68432c596f68314f185a0ca49f3"}
Sep 11 16:18:12.971585 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.971498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p7mzb" event={"ID":"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912","Type":"ContainerStarted","Data":"7ea71b9bf088d5c4f7df7020e147b85e0488b7d28613449261596cb4f2d5949d"}
Sep 11 16:18:12.973323 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.973293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerStarted","Data":"407ba99b7d83d18165024ecd6eb5d39f734429c8789362af1d049e2f363abe0e"}
Sep 11 16:18:12.977418 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.977329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl" event={"ID":"baa82fcf-1fb5-428e-9de2-03e32dec5017","Type":"ContainerStarted","Data":"5d32b82854463aa24e3fa7c93871270769978c32639e7be4eabedbf77f9daeda"}
Sep 11 16:18:12.980592 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.980505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6gfq" event={"ID":"3da31bc3-f199-4904-9c3d-de3d60ff4815","Type":"ContainerStarted","Data":"88c23d657333934fc328ea90110bb2faa4a915ea2d3a767339186b719f8868d8"}
Sep 11 16:18:12.987743 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.987664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bg5xb" event={"ID":"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6","Type":"ContainerStarted","Data":"b75205d7a1d47846bf830c70095e9402af3786c5a64572274d72f76309805320"}
Sep 11 16:18:12.993807 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.993698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"6837c9bb02085f17e0dde4c6393d3f99801c75b4592bd7efc37f6d70800cb909"}
Sep 11 16:18:12.997540 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:12.997506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b779c" event={"ID":"a0f6a39a-b4d2-420b-86d6-6535644a7dce","Type":"ContainerStarted","Data":"80cf0ede3fab888e14b2c7c753e627c458b4ef58b187d15224562bba11a2e956"}
Sep 11 16:18:13.007318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.007020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mvn6l" event={"ID":"815ae807-3454-424e-8beb-77f2fdce09bf","Type":"ContainerStarted","Data":"88d05acf5a91912670582d142c156d6eecd67d3f1b3024ef2af6a2cfbd843b95"}
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.507716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.507780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.507808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.507887    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.507916    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.507930    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.507930    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.507988    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:15.507969862 +0000 UTC m=+40.125939347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.508026    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.508028    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:15.507997896 +0000 UTC m=+40.125967377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:13.508142 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.508095    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:15.508078556 +0000 UTC m=+40.126048039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:13.589178 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.589147    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:18:13.830105 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.829974    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:13:11 +0000 UTC" deadline="2028-06-06 14:47:36.114172212 +0000 UTC"
Sep 11 16:18:13.830105 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.830040    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15214h29m22.28413591s"
Sep 11 16:18:13.893405 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.893086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:13.893405 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.893204    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:13.893642 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.893605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:13.893954 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.893697    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:13.893954 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:13.893766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:13.893954 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:13.893856    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:15.108349 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.108315    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hlnft"]
Sep 11 16:18:15.110357 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.110331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.113688 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.113663    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:18:15.114220 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.114201    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:18:15.114474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.114457    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6vtlj\""
Sep 11 16:18:15.114656 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.114639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:18:15.114877 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.114860    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:18:15.115087 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.115071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:18:15.115618 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.115600    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:18:15.221664 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.221839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-sys\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.221839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-root\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.221839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.221839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-textfile\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.221839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-wtmp\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.221839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef54c37d-b09a-4c08-b617-a286b8131213-metrics-client-ca\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.222036 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221853    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2bcv\" (UniqueName: \"kubernetes.io/projected/ef54c37d-b09a-4c08-b617-a286b8131213-kube-api-access-s2bcv\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.222036 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.221877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-tls\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322448 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-sys\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-root\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-textfile\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-wtmp\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef54c37d-b09a-4c08-b617-a286b8131213-metrics-client-ca\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2bcv\" (UniqueName: \"kubernetes.io/projected/ef54c37d-b09a-4c08-b617-a286b8131213-kube-api-access-s2bcv\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.322956 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.322827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-tls\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.323586 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.323560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-wtmp\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.323884 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.323813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-textfile\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.323884 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.323877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-sys\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.324155 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.323974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-accelerators-collector-config\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.324155 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.324070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ef54c37d-b09a-4c08-b617-a286b8131213-root\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.324524 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.324501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef54c37d-b09a-4c08-b617-a286b8131213-metrics-client-ca\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.326628 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.326598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.328074 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.328032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ef54c37d-b09a-4c08-b617-a286b8131213-node-exporter-tls\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.341098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.341039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2bcv\" (UniqueName: \"kubernetes.io/projected/ef54c37d-b09a-4c08-b617-a286b8131213-kube-api-access-s2bcv\") pod \"node-exporter-hlnft\" (UID: \"ef54c37d-b09a-4c08-b617-a286b8131213\") " pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.426576 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.426492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hlnft"
Sep 11 16:18:15.524513 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.524472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:15.524706 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.524545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:15.524706 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.524574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:15.524816 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.524710    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:15.524816 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.524803    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:19.524780612 +0000 UTC m=+44.142750092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:15.525253 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.525231    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:15.525338 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.525258    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:15.525338 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.525271    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:15.525338 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.525323    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:19.525307608 +0000 UTC m=+44.143277090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:15.525494 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.525386    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:15.525494 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.525421    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:19.525412095 +0000 UTC m=+44.143381575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:15.895266 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.895241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:15.895462 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.895437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:15.895641 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:15.895624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:15.895736 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.895705    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:15.895831 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.895796    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:15.895958 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:15.895942    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:17.890467 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:17.890434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:17.890901 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:17.890434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:17.890901 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:17.890583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:17.890901 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:17.890575    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:17.890901 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:17.890720    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:17.890901 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:17.890809    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:19.561678 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:19.561641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:19.562139 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:19.561701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:19.562139 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:19.561727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:19.562139 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.561857    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:19.562139 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.561918    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.561899976 +0000 UTC m=+52.179869480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:19.562393 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.562374    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:19.562455 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.562398    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:19.562455 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.562411    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:19.562551 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.562457    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.562442739 +0000 UTC m=+52.180412221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:19.562551 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.562524    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:19.562640 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.562559    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.56254661 +0000 UTC m=+52.180516092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:19.893070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:19.892278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:19.893070 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.892408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:19.893070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:19.892783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:19.893070 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.892893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:19.893070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:19.892938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:19.893070 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:19.893002    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:21.890401 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:21.890364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:21.890881 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:21.890364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:21.890881 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:21.890503    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:21.890881 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:21.890364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:21.890881 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:21.890564    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:21.890881 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:21.890674    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:23.890143 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:23.890102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:23.890143 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:23.890130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:23.890668 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:23.890245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:23.890668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:23.890286    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:23.890668 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:23.890407    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:23.890668 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:23.890493    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:25.891698 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:25.891659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:25.892211 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:25.891749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:25.892211 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:25.891869    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:25.892211 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:25.891924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:25.892211 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:25.891987    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:25.892211 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:25.891749    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:27.627894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:27.627849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:27.627933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:27.627976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628041    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628092    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628124    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:43.628129642 +0000 UTC m=+68.246099121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628146    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628160    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628161    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:43.628153136 +0000 UTC m=+68.246122615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:27.628365 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.628202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:43.628186755 +0000 UTC m=+68.246156246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:27.890934 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:27.890856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:27.890934 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:27.890884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:27.891177 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.890967    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:27.891177 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:27.890856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:27.891177 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.891081    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:27.891177 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:27.891166    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:29.890441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:29.890230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:29.891058 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:29.890545    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:29.891058 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:29.890625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:29.891058 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:29.890751    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:29.891058 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:29.890810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:29.891058 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:29.890863    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:30.054128 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.054093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6gfq" event={"ID":"3da31bc3-f199-4904-9c3d-de3d60ff4815","Type":"ContainerStarted","Data":"1e96011d16cc22529431ab84fa5d6aa880f6eab3053ef81789c69b3b03accb2a"}
Sep 11 16:18:30.055555 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.055533    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlnft" event={"ID":"ef54c37d-b09a-4c08-b617-a286b8131213","Type":"ContainerStarted","Data":"db1b1ba86fc76751ff90b0341f945082d7af8f2737c26e9bf9c9b07e24d9a668"}
Sep 11 16:18:30.057748 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.057731    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_ec3255f401d21cf30d3c48f303e873f0/setup/1.log"
Sep 11 16:18:30.057843 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.057770    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec3255f401d21cf30d3c48f303e873f0" containerID="a122a9ec0c66f4af59965795ee46fa9a5179c3e9f80e5d30441f3a9c0012de91" exitCode=0
Sep 11 16:18:30.057896 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.057843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerDied","Data":"a122a9ec0c66f4af59965795ee46fa9a5179c3e9f80e5d30441f3a9c0012de91"}
Sep 11 16:18:30.057947 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.057914    2578 scope.go:117] "RemoveContainer" containerID="a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a"
Sep 11 16:18:30.058229 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.058212    2578 scope.go:117] "RemoveContainer" containerID="a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a"
Sep 11 16:18:30.062197 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.062170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"b16c3ac08a36770cd2de6d2da696dd2e83a0fe4ff5e1a05554511458a19212e8"}
Sep 11 16:18:30.062300 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.062255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"c463fa37d0c3afb47314e796c610c1c0bd17b399efb2d76ec62fd3c1a0b63a5c"}
Sep 11 16:18:30.062391 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.062309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"eed553dbccb2214b74e9cdeeceefe9b2701a0ae0735fbcd327d3634ef0f7fada"}
Sep 11 16:18:30.062391 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.062355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"d1bd898c4c6713c5102c9223cc937701d84bb7d34ef4d3f0747af3826f49cc94"}
Sep 11 16:18:30.062477 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.062404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"b96e186f441e284ad367cc0b0b0cedd7dcfb745c68265f050edaee500b429aeb"}
Sep 11 16:18:30.063837 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.063814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b779c" event={"ID":"a0f6a39a-b4d2-420b-86d6-6535644a7dce","Type":"ContainerStarted","Data":"5c7281d5af473b8192c233821ccb04a902681ffa35a343813f263b9f9815877b"}
Sep 11 16:18:30.069466 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.065728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mvn6l" event={"ID":"815ae807-3454-424e-8beb-77f2fdce09bf","Type":"ContainerStarted","Data":"df6ba51aa86d2e33089a5acc45ee3f3d436f8d834418e14c81abb1cc749adca5"}
Sep 11 16:18:30.075027 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:30.071872    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator_ec3255f401d21cf30d3c48f303e873f0_1 in pod sandbox da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0 from index: no such id: 'a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a'" containerID="a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a"
Sep 11 16:18:30.075027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.071919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal_openshift-machine-config-operator_ec3255f401d21cf30d3c48f303e873f0_1 in pod sandbox da09417920dffdd51188eb3a6f87bc49d39ebc2c8f3ca184f4f222f06698ebb0 from index: no such id: 'a422f38a41da22b07df5546e874cf1ce74d813f1b550d7735e273cea472d726a'"
Sep 11 16:18:30.075027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.074218    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k6gfq" podStartSLOduration=10.765188786 podStartE2EDuration="28.074196567s" totalImagesPullingTime="17.309007781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.072985288 +0000 UTC m=+54.690954790" watchObservedRunningTime="2026-09-11 16:18:30.074196567 +0000 UTC m=+54.692166070"
Sep 11 16:18:30.078668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.078559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c5vjm" event={"ID":"fc1cdc55-6865-430d-8810-8d306526177f","Type":"ContainerStarted","Data":"3b98d2a341467c35d495b9bd3c6fc6b70a6ef96e836de8b14c8eef64aee6d841"}
Sep 11 16:18:30.080998 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.080939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p7mzb" event={"ID":"f0e6493f-bb8e-4c87-8f2e-9bb5a6861912","Type":"ContainerStarted","Data":"dff776fd8072442f599b42679f591bf841f641c69cd537aa14dec2772fa4e480"}
Sep 11 16:18:30.083302 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.083276    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0" containerID="55379a557ae84fd22aa02be93d51f55247079e1e4bf5c1496d5c810cac4095c4" exitCode=0
Sep 11 16:18:30.083419 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.083379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerDied","Data":"55379a557ae84fd22aa02be93d51f55247079e1e4bf5c1496d5c810cac4095c4"}
Sep 11 16:18:30.085547 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.085522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl" event={"ID":"baa82fcf-1fb5-428e-9de2-03e32dec5017","Type":"ContainerStarted","Data":"fdcf803ed4195e731d6d4dd57d6801c4bcde7c77c5364c6ca148034a20f40e5c"}
Sep 11 16:18:30.118789 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.118730    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mvn6l" podStartSLOduration=10.594551545 podStartE2EDuration="28.11871154s" totalImagesPullingTime="17.524159995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.095980663 +0000 UTC m=+54.713950165" watchObservedRunningTime="2026-09-11 16:18:30.11871154 +0000 UTC m=+54.736681041"
Sep 11 16:18:30.135342 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.135294    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b779c" podStartSLOduration=10.825458154 podStartE2EDuration="28.135278594s" totalImagesPullingTime="17.30982044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.135087351 +0000 UTC m=+54.753056852" watchObservedRunningTime="2026-09-11 16:18:30.135278594 +0000 UTC m=+54.753248094"
Sep 11 16:18:30.185427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.185262    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p7mzb" podStartSLOduration=10.862432411 podStartE2EDuration="28.185246495s" totalImagesPullingTime="17.322814084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.1835522 +0000 UTC m=+54.801521700" watchObservedRunningTime="2026-09-11 16:18:30.185246495 +0000 UTC m=+54.803215996"
Sep 11 16:18:30.951968 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:30.951945    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:18:31.090302 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.090257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" event={"ID":"ec3255f401d21cf30d3c48f303e873f0","Type":"ContainerStarted","Data":"6d4b2ef4adaf2d77775e8cb8b76563cf96f31cda16f4429ec2dc1eb0bf8a51d5"}
Sep 11 16:18:31.094363 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.094329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"0b8a9ed1fa62fbf9d44c09d1f39c6ec85b8534cd83b281c9268f823358535f0d"}
Sep 11 16:18:31.096372 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.096298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl" event={"ID":"baa82fcf-1fb5-428e-9de2-03e32dec5017","Type":"ContainerStarted","Data":"ed0e757fb185ba7813de8946674a59589390b390e90d65eadfae5b973cbb1060"}
Sep 11 16:18:31.098223 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.098198    2578 generic.go:356] "Generic (PLEG): container finished" podID="ef54c37d-b09a-4c08-b617-a286b8131213" containerID="2e7f3aaa93bf4a2e33f19628474ea717e1a299796f2289fff89a3222a6c587d2" exitCode=0
Sep 11 16:18:31.098349 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.098304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlnft" event={"ID":"ef54c37d-b09a-4c08-b617-a286b8131213","Type":"ContainerDied","Data":"2e7f3aaa93bf4a2e33f19628474ea717e1a299796f2289fff89a3222a6c587d2"}
Sep 11 16:18:31.107350 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.107289    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c5vjm" podStartSLOduration=11.85279785 podStartE2EDuration="29.107268303s" totalImagesPullingTime="17.254470453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:30.212283992 +0000 UTC m=+54.830253492" watchObservedRunningTime="2026-09-11 16:18:31.107268303 +0000 UTC m=+55.725237820"
Sep 11 16:18:31.107561 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.107525    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.ec2.internal" podStartSLOduration=20.107516597 podStartE2EDuration="20.107516597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:31.107038886 +0000 UTC m=+55.725008386" watchObservedRunningTime="2026-09-11 16:18:31.107516597 +0000 UTC m=+55.725486103"
Sep 11 16:18:31.874278 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.874182    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:18:30.951964854Z","UUID":"233235b5-b4ff-4ca1-806b-292a3bf6e8e3","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:18:31.875802 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.875781    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:18:31.875944 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.875819    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:18:31.890887 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.890853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:31.891069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.890853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:31.891069 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:31.890969    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:31.891069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:31.890868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:31.891251 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:31.891084    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:31.891251 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:31.891166    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:32.102350 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.102254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bg5xb" event={"ID":"39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6","Type":"ContainerStarted","Data":"9d738c4fbbb3fea584fb4c41306a3f4dcb8634470c47aba6567014d4a4df2a94"}
Sep 11 16:18:32.104830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.104791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlnft" event={"ID":"ef54c37d-b09a-4c08-b617-a286b8131213","Type":"ContainerStarted","Data":"d45635407708d01969e43429073f9de3e294efc8ca08e3d48054d8e3c3ef4bc7"}
Sep 11 16:18:32.104977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.104855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hlnft" event={"ID":"ef54c37d-b09a-4c08-b617-a286b8131213","Type":"ContainerStarted","Data":"dd684df60c6947b8250c36701230bdcaf46099ae803ee987f23fe5f917ce1241"}
Sep 11 16:18:32.107194 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.107169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl" event={"ID":"baa82fcf-1fb5-428e-9de2-03e32dec5017","Type":"ContainerStarted","Data":"6859d0ebfd78bd5a81c3c158936f1019a708e9f3ebd6b4cc60d4ec456f442a09"}
Sep 11 16:18:32.118732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.118693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:32.119345 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.119317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:32.119490 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.119383    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bg5xb" podStartSLOduration=12.769572372 podStartE2EDuration="30.119357178s" totalImagesPullingTime="17.349784806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:32.118993679 +0000 UTC m=+56.736963186" watchObservedRunningTime="2026-09-11 16:18:32.119357178 +0000 UTC m=+56.737326678"
Sep 11 16:18:32.137732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.137670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hlnft" podStartSLOduration=16.102780647 podStartE2EDuration="17.137652218s" totalImagesPullingTime="943.931865ms" totalInitContainerRuntime="90.939706ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:32.137517912 +0000 UTC m=+56.755487423" watchObservedRunningTime="2026-09-11 16:18:32.137652218 +0000 UTC m=+56.755621719"
Sep 11 16:18:32.156250 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:32.156195    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl8gl" podStartSLOduration=11.057758945 podStartE2EDuration="30.156176529s" totalImagesPullingTime="19.098417584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:32.155481664 +0000 UTC m=+56.773451170" watchObservedRunningTime="2026-09-11 16:18:32.156176529 +0000 UTC m=+56.774146029"
Sep 11 16:18:33.118016 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:33.117977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"4de77a0d50d665bbaefc9896e44cd39cafeafdae297dd69d55c07193653be82a"}
Sep 11 16:18:33.118925 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:33.118901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:33.119062 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:33.118969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k6gfq"
Sep 11 16:18:33.891029 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:33.890993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:33.891228 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:33.890995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:33.891228 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:33.891146    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:33.891228 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:33.891215    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:33.891392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:33.891001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:33.891392 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:33.891297    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:35.126058 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.125799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" event={"ID":"02b9217e-d171-4f08-95fa-90bd09c6675a","Type":"ContainerStarted","Data":"920db98b292ac64b17dc8de302c74080f4ca0dafcf56b8b07a9b0e102f67800b"}
Sep 11 16:18:35.126915 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.126109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:35.126915 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.126158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:35.128324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.128292    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0" containerID="307109df41ecd28c3282debd7e2b9a6190a0753ff5215c5392f5c89daea76435" exitCode=0
Sep 11 16:18:35.128440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.128351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerDied","Data":"307109df41ecd28c3282debd7e2b9a6190a0753ff5215c5392f5c89daea76435"}
Sep 11 16:18:35.143150 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.143125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:35.158440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.158392    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x" podStartSLOduration=14.835858884 podStartE2EDuration="32.158376296s" totalImagesPullingTime="17.322517412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:35.157698816 +0000 UTC m=+59.775668316" watchObservedRunningTime="2026-09-11 16:18:35.158376296 +0000 UTC m=+59.776345797"
Sep 11 16:18:35.892147 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.891909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:35.892354 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.892008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:35.892354 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:35.892245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:35.892354 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:35.892323    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:35.892354 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:35.892036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:35.892579 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:35.892409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:36.131745 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.131714    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:36.147466 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.147374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:18:36.479464 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.479434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5q6j"]
Sep 11 16:18:36.479600 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.479591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:36.479717 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:36.479694    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:36.483453 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.483428    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gctq"]
Sep 11 16:18:36.483566 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.483542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:36.483683 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:36.483637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:36.484290 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.484265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s2ks4"]
Sep 11 16:18:36.484398 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:36.484381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:36.484529 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:36.484506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:37.136212 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:37.136174    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0" containerID="db5bc0f52577ee2464a91beba014c155fd492d06628dff4d5b3f12dd531051f9" exitCode=0
Sep 11 16:18:37.136643 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:37.136252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerDied","Data":"db5bc0f52577ee2464a91beba014c155fd492d06628dff4d5b3f12dd531051f9"}
Sep 11 16:18:37.890166 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:37.890130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:37.890352 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:37.890253    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:37.890352 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:37.890310    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:37.890464 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:37.890402    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:38.890399 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:38.890311    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:38.890843 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:38.890449    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:39.144305 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:39.144018    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0" containerID="4426d2b1d8fcb7a9ffbe91a5eece1be89cebf3fbd6e58fa030b3be0be4022e66" exitCode=0
Sep 11 16:18:39.144305 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:39.144094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerDied","Data":"4426d2b1d8fcb7a9ffbe91a5eece1be89cebf3fbd6e58fa030b3be0be4022e66"}
Sep 11 16:18:39.891580 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:39.891550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:39.891973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:39.891592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:39.891973 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:39.891724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:39.892252 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:39.892228    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:40.890435 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:40.890403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:40.890659 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:40.890531    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:41.890847 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:41.890805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:41.891358 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:41.890821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:41.891358 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:41.890947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p5q6j" podUID="179f2ddc-df25-46bf-b640-94826d4f434d"
Sep 11 16:18:41.891358 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:41.891033    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2ks4" podUID="51b154f6-27bb-4554-b269-0cfa1de4bda7"
Sep 11 16:18:42.890015 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:42.889981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:42.890207 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:42.890123    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8gctq" podUID="e2f260be-4aab-4a68-87d2-35cf264b3d1d"
Sep 11 16:18:43.654503 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.654468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.654515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.654541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654633    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654640    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654674    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654687    2578 projected.go:196] Error preparing data for projected volume kube-api-access-rdjnn for pod openshift-network-diagnostics/network-check-target-8gctq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654691    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs podName:51b154f6-27bb-4554-b269-0cfa1de4bda7 nodeName:}" failed. No retries permitted until 2026-09-11 16:19:15.654678185 +0000 UTC m=+100.272647677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs") pod "network-metrics-daemon-s2ks4" (UID: "51b154f6-27bb-4554-b269-0cfa1de4bda7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654729    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn podName:e2f260be-4aab-4a68-87d2-35cf264b3d1d nodeName:}" failed. No retries permitted until 2026-09-11 16:19:15.654715033 +0000 UTC m=+100.272684516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rdjnn" (UniqueName: "kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn") pod "network-check-target-8gctq" (UID: "e2f260be-4aab-4a68-87d2-35cf264b3d1d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654637    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:43.654981 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:18:43.654776    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret podName:179f2ddc-df25-46bf-b640-94826d4f434d nodeName:}" failed. No retries permitted until 2026-09-11 16:19:15.654764509 +0000 UTC m=+100.272733990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret") pod "global-pull-secret-syncer-p5q6j" (UID: "179f2ddc-df25-46bf-b640-94826d4f434d") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:18:43.678502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.678480    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-5.ec2.internal" event="NodeReady"
Sep 11 16:18:43.678636 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.678617    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:18:43.746783 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.746744    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g5pxx"]
Sep 11 16:18:43.764000 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.763964    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vsb48"]
Sep 11 16:18:43.764178 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.764095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:43.767621 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.767596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:18:43.767756 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.767690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:18:43.767756 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.767706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:18:43.767756 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.767739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gnx8w\""
Sep 11 16:18:43.768127 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.768101    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:18:43.786092 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.786017    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g5pxx"]
Sep 11 16:18:43.786209 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.786092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vsb48"]
Sep 11 16:18:43.786209 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.786203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:43.795250 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.795225    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:18:43.795373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.795316    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:18:43.795585 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.795571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hnfgd\""
Sep 11 16:18:43.797220 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.797205    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:18:43.852226 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.852199    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g55bs"]
Sep 11 16:18:43.875347 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.875302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g55bs"]
Sep 11 16:18:43.875522 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.875411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:43.878355 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.878314    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lsfrm\""
Sep 11 16:18:43.878355 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.878344    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:18:43.878703 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.878688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:18:43.889980 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.889957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:18:43.890217 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.890199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:18:43.892874 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.892853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:18:43.892995 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.892936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:18:43.893111 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.893095    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn9xf\""
Sep 11 16:18:43.956697 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc27f\" (UniqueName: \"kubernetes.io/projected/fc7eab1c-9b8f-4a35-86b8-832abe802552-kube-api-access-vc27f\") pod \"ingress-canary-vsb48\" (UID: \"fc7eab1c-9b8f-4a35-86b8-832abe802552\") " pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:43.956894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc7eab1c-9b8f-4a35-86b8-832abe802552-cert\") pod \"ingress-canary-vsb48\" (UID: \"fc7eab1c-9b8f-4a35-86b8-832abe802552\") " pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:43.956894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxdl\" (UniqueName: \"kubernetes.io/projected/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-kube-api-access-qjxdl\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:43.956894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:43.957092 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-crio-socket\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:43.957092 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-data-volume\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:43.957092 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:43.956995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.057959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvwx\" (UniqueName: \"kubernetes.io/projected/febf625b-e5c1-417f-98cf-3c3ef5988d5a-kube-api-access-dlvwx\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.058069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/febf625b-e5c1-417f-98cf-3c3ef5988d5a-config-volume\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.058311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-crio-socket\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-data-volume\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febf625b-e5c1-417f-98cf-3c3ef5988d5a-tmp-dir\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.058311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc27f\" (UniqueName: \"kubernetes.io/projected/fc7eab1c-9b8f-4a35-86b8-832abe802552-kube-api-access-vc27f\") pod \"ingress-canary-vsb48\" (UID: \"fc7eab1c-9b8f-4a35-86b8-832abe802552\") " pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:44.058514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-crio-socket\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/febf625b-e5c1-417f-98cf-3c3ef5988d5a-metrics-tls\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.058514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc7eab1c-9b8f-4a35-86b8-832abe802552-cert\") pod \"ingress-canary-vsb48\" (UID: \"fc7eab1c-9b8f-4a35-86b8-832abe802552\") " pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:44.058514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxdl\" (UniqueName: \"kubernetes.io/projected/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-kube-api-access-qjxdl\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-data-volume\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.058718 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.058700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.062187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.062169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.062319 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.062303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc7eab1c-9b8f-4a35-86b8-832abe802552-cert\") pod \"ingress-canary-vsb48\" (UID: \"fc7eab1c-9b8f-4a35-86b8-832abe802552\") " pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:44.068641 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.068610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc27f\" (UniqueName: \"kubernetes.io/projected/fc7eab1c-9b8f-4a35-86b8-832abe802552-kube-api-access-vc27f\") pod \"ingress-canary-vsb48\" (UID: \"fc7eab1c-9b8f-4a35-86b8-832abe802552\") " pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:44.068735 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.068615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxdl\" (UniqueName: \"kubernetes.io/projected/3b5847d0-27fe-49e6-bb07-127e2fd8b4b6-kube-api-access-qjxdl\") pod \"insights-runtime-extractor-g5pxx\" (UID: \"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6\") " pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.073535 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.073517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g5pxx"
Sep 11 16:18:44.145514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.145479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vsb48"
Sep 11 16:18:44.158799 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.158769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/febf625b-e5c1-417f-98cf-3c3ef5988d5a-metrics-tls\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.158962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.158816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvwx\" (UniqueName: \"kubernetes.io/projected/febf625b-e5c1-417f-98cf-3c3ef5988d5a-kube-api-access-dlvwx\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.158962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.158858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/febf625b-e5c1-417f-98cf-3c3ef5988d5a-config-volume\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.158962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.158896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febf625b-e5c1-417f-98cf-3c3ef5988d5a-tmp-dir\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.159337 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.159312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febf625b-e5c1-417f-98cf-3c3ef5988d5a-tmp-dir\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.159658 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.159636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/febf625b-e5c1-417f-98cf-3c3ef5988d5a-config-volume\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.161691 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.161661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/febf625b-e5c1-417f-98cf-3c3ef5988d5a-metrics-tls\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.162163 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.162136    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0" containerID="2a4851cc47e64dcd1a10373de02c12b73b25e51a90eb647666dc6aa64e31f149" exitCode=0
Sep 11 16:18:44.162242 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.162217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerDied","Data":"2a4851cc47e64dcd1a10373de02c12b73b25e51a90eb647666dc6aa64e31f149"}
Sep 11 16:18:44.176189 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.175952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvwx\" (UniqueName: \"kubernetes.io/projected/febf625b-e5c1-417f-98cf-3c3ef5988d5a-kube-api-access-dlvwx\") pod \"dns-default-g55bs\" (UID: \"febf625b-e5c1-417f-98cf-3c3ef5988d5a\") " pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.183483 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.183331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:44.412459 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.412434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g5pxx"]
Sep 11 16:18:44.464943 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.464905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vsb48"]
Sep 11 16:18:44.502026 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.501993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g55bs"]
Sep 11 16:18:44.505776 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:44.505729    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5847d0_27fe_49e6_bb07_127e2fd8b4b6.slice/crio-1a682b6148e92c039d7516e8798906f019ee3e87e1a931db32082e7cebfa3adc WatchSource:0}: Error finding container 1a682b6148e92c039d7516e8798906f019ee3e87e1a931db32082e7cebfa3adc: Status 404 returned error can't find the container with id 1a682b6148e92c039d7516e8798906f019ee3e87e1a931db32082e7cebfa3adc
Sep 11 16:18:44.506557 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:44.506535    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7eab1c_9b8f_4a35_86b8_832abe802552.slice/crio-48e6f5b92a1de5f27083b081894d045c7683c8c6cf98d1903a47e52901b1e6ce WatchSource:0}: Error finding container 48e6f5b92a1de5f27083b081894d045c7683c8c6cf98d1903a47e52901b1e6ce: Status 404 returned error can't find the container with id 48e6f5b92a1de5f27083b081894d045c7683c8c6cf98d1903a47e52901b1e6ce
Sep 11 16:18:44.507301 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:18:44.507279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebf625b_e5c1_417f_98cf_3c3ef5988d5a.slice/crio-504a2cbddff27861fd971e0c5e6879f38131f4af9cdd13bc8d805fc75b11cec5 WatchSource:0}: Error finding container 504a2cbddff27861fd971e0c5e6879f38131f4af9cdd13bc8d805fc75b11cec5: Status 404 returned error can't find the container with id 504a2cbddff27861fd971e0c5e6879f38131f4af9cdd13bc8d805fc75b11cec5
Sep 11 16:18:44.890266 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.890176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:18:44.893343 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.893323    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:18:44.893427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.893345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mk9m2\""
Sep 11 16:18:44.894410 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:44.894397    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:18:45.170192 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:45.170097    2578 generic.go:356] "Generic (PLEG): container finished" podID="a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0" containerID="5c8fab6403c85a43e58e6d545fa3298ced7d07d11e447e2a973841fb7c8be994" exitCode=0
Sep 11 16:18:45.170192 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:45.170155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerDied","Data":"5c8fab6403c85a43e58e6d545fa3298ced7d07d11e447e2a973841fb7c8be994"}
Sep 11 16:18:45.171768 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:45.171732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g55bs" event={"ID":"febf625b-e5c1-417f-98cf-3c3ef5988d5a","Type":"ContainerStarted","Data":"504a2cbddff27861fd971e0c5e6879f38131f4af9cdd13bc8d805fc75b11cec5"}
Sep 11 16:18:45.174026 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:45.173999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vsb48" event={"ID":"fc7eab1c-9b8f-4a35-86b8-832abe802552","Type":"ContainerStarted","Data":"48e6f5b92a1de5f27083b081894d045c7683c8c6cf98d1903a47e52901b1e6ce"}
Sep 11 16:18:45.177473 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:45.177448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5pxx" event={"ID":"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6","Type":"ContainerStarted","Data":"1e3960b0260a4ff077924e752e2dc5f95df41435f5a438eccaadfb325c56a416"}
Sep 11 16:18:45.177571 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:45.177536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5pxx" event={"ID":"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6","Type":"ContainerStarted","Data":"1a682b6148e92c039d7516e8798906f019ee3e87e1a931db32082e7cebfa3adc"}
Sep 11 16:18:46.183507 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:46.183219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5pxx" event={"ID":"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6","Type":"ContainerStarted","Data":"e7069c79770a9e1e8534d99eca67f101ec8c8081245cc9b335315d7b198f168f"}
Sep 11 16:18:46.187689 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:46.187662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86z74" event={"ID":"a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0","Type":"ContainerStarted","Data":"136d1e247619a8669453832a7724e4e41f059f80db2f371ce93a83e22387b7cb"}
Sep 11 16:18:46.228764 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:46.228699    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86z74" podStartSLOduration=15.788869497 podStartE2EDuration="44.228677046s" totalImagesPullingTime="28.439807549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:46.22640549 +0000 UTC m=+70.844374989" watchObservedRunningTime="2026-09-11 16:18:46.228677046 +0000 UTC m=+70.846646547"
Sep 11 16:18:48.196862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:48.196826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g55bs" event={"ID":"febf625b-e5c1-417f-98cf-3c3ef5988d5a","Type":"ContainerStarted","Data":"f69ac24e14860e6343be58f23ed59667cc41f2e3836c6798a99e1fb8d6cd0e0d"}
Sep 11 16:18:48.197384 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:48.196903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g55bs" event={"ID":"febf625b-e5c1-417f-98cf-3c3ef5988d5a","Type":"ContainerStarted","Data":"098a3aff745e313564c8a09702e59fab933c154dc385d7f5a22a0764799cc17d"}
Sep 11 16:18:48.197384 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:48.196942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g55bs"
Sep 11 16:18:48.198507 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:48.198490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vsb48" event={"ID":"fc7eab1c-9b8f-4a35-86b8-832abe802552","Type":"ContainerStarted","Data":"369e5ab3e1631134af83ad1de6c2fa1726559659a8c6bc9d3818e1acc8de850b"}
Sep 11 16:18:48.218678 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:48.218622    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g55bs" podStartSLOduration=1.994277416 podStartE2EDuration="5.218605583s" totalImagesPullingTime="3.224328167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:48.217323636 +0000 UTC m=+72.835293137" watchObservedRunningTime="2026-09-11 16:18:48.218605583 +0000 UTC m=+72.836575079"
Sep 11 16:18:48.235004 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:48.234947    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vsb48" podStartSLOduration=2.012209067 podStartE2EDuration="5.234928301s" totalImagesPullingTime="3.222719234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:48.234127355 +0000 UTC m=+72.852096855" watchObservedRunningTime="2026-09-11 16:18:48.234928301 +0000 UTC m=+72.852897866"
Sep 11 16:18:49.203961 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:49.203926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5pxx" event={"ID":"3b5847d0-27fe-49e6-bb07-127e2fd8b4b6","Type":"ContainerStarted","Data":"b0fc3f3af8aeed68eab753ce05f552d4bcd8ab00316062ee4a0fd57ed69e3362"}
Sep 11 16:18:49.226514 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:49.226466    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g5pxx" podStartSLOduration=2.142792396 podStartE2EDuration="6.226448712s" totalImagesPullingTime="4.083656316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:49.225141531 +0000 UTC m=+73.843111032" watchObservedRunningTime="2026-09-11 16:18:49.226448712 +0000 UTC m=+73.844418203"
Sep 11 16:18:58.206279 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:18:58.206172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g55bs"
Sep 11 16:19:08.150763 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:08.150739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqb2x"
Sep 11 16:19:15.694143 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.694091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:19:15.694143 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.694150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:19:15.694770 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.694185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:19:15.698187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.698153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51b154f6-27bb-4554-b269-0cfa1de4bda7-metrics-certs\") pod \"network-metrics-daemon-s2ks4\" (UID: \"51b154f6-27bb-4554-b269-0cfa1de4bda7\") " pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:19:15.698309 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.698220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/179f2ddc-df25-46bf-b640-94826d4f434d-original-pull-secret\") pod \"global-pull-secret-syncer-p5q6j\" (UID: \"179f2ddc-df25-46bf-b640-94826d4f434d\") " pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:19:15.698396 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.698381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjnn\" (UniqueName: \"kubernetes.io/projected/e2f260be-4aab-4a68-87d2-35cf264b3d1d-kube-api-access-rdjnn\") pod \"network-check-target-8gctq\" (UID: \"e2f260be-4aab-4a68-87d2-35cf264b3d1d\") " pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:19:15.699023 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.699007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2ks4"
Sep 11 16:19:15.704077 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.704027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p5q6j"
Sep 11 16:19:15.799482 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.799448    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:19:15.975774 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.975684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p5q6j"]
Sep 11 16:19:15.976733 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:15.976706    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s2ks4"]
Sep 11 16:19:15.980401 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:19:15.980351    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b154f6_27bb_4554_b269_0cfa1de4bda7.slice/crio-8df06be2a88113b48f2cd8108b9d9ea351d0ed937d32b09104568cd63d03ebd6 WatchSource:0}: Error finding container 8df06be2a88113b48f2cd8108b9d9ea351d0ed937d32b09104568cd63d03ebd6: Status 404 returned error can't find the container with id 8df06be2a88113b48f2cd8108b9d9ea351d0ed937d32b09104568cd63d03ebd6
Sep 11 16:19:15.980635 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:19:15.980596    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179f2ddc_df25_46bf_b640_94826d4f434d.slice/crio-3c738c7f29ffd65a481b58ae0765a3c57abb52076299b724e67f071d08c2c366 WatchSource:0}: Error finding container 3c738c7f29ffd65a481b58ae0765a3c57abb52076299b724e67f071d08c2c366: Status 404 returned error can't find the container with id 3c738c7f29ffd65a481b58ae0765a3c57abb52076299b724e67f071d08c2c366
Sep 11 16:19:16.068341 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:16.068312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8gctq"]
Sep 11 16:19:16.071218 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:19:16.071196    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f260be_4aab_4a68_87d2_35cf264b3d1d.slice/crio-43ae5a81b0b88d1a8bdb5ae219a7c4b5b2c7289e305a7074033abfa98775e915 WatchSource:0}: Error finding container 43ae5a81b0b88d1a8bdb5ae219a7c4b5b2c7289e305a7074033abfa98775e915: Status 404 returned error can't find the container with id 43ae5a81b0b88d1a8bdb5ae219a7c4b5b2c7289e305a7074033abfa98775e915
Sep 11 16:19:16.290573 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:16.290487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2ks4" event={"ID":"51b154f6-27bb-4554-b269-0cfa1de4bda7","Type":"ContainerStarted","Data":"8df06be2a88113b48f2cd8108b9d9ea351d0ed937d32b09104568cd63d03ebd6"}
Sep 11 16:19:16.292169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:16.292141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gctq" event={"ID":"e2f260be-4aab-4a68-87d2-35cf264b3d1d","Type":"ContainerStarted","Data":"43ae5a81b0b88d1a8bdb5ae219a7c4b5b2c7289e305a7074033abfa98775e915"}
Sep 11 16:19:16.293274 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:16.293247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5q6j" event={"ID":"179f2ddc-df25-46bf-b640-94826d4f434d","Type":"ContainerStarted","Data":"3c738c7f29ffd65a481b58ae0765a3c57abb52076299b724e67f071d08c2c366"}
Sep 11 16:19:17.301180 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:17.301136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2ks4" event={"ID":"51b154f6-27bb-4554-b269-0cfa1de4bda7","Type":"ContainerStarted","Data":"c83923139f8628d84add6714e33a39af1093428314b70e21302143ab161e2a5d"}
Sep 11 16:19:17.301624 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:17.301206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2ks4" event={"ID":"51b154f6-27bb-4554-b269-0cfa1de4bda7","Type":"ContainerStarted","Data":"e742d333167ce89f1726f9564c1bfef64834a218a65c7341a67e1c3678fae7a9"}
Sep 11 16:19:17.319311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:17.319247    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s2ks4" podStartSLOduration=74.473196973 podStartE2EDuration="1m15.319222876s" totalImagesPullingTime="846.025903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:17.318150462 +0000 UTC m=+101.936119964" watchObservedRunningTime="2026-09-11 16:19:17.319222876 +0000 UTC m=+101.937192382"
Sep 11 16:19:21.318207 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:21.318090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8gctq" event={"ID":"e2f260be-4aab-4a68-87d2-35cf264b3d1d","Type":"ContainerStarted","Data":"a62109f2b0eb71be862d16a06f398a54104cbe28eaa34d1afbf104b93f1e662c"}
Sep 11 16:19:21.318737 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:21.318213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:19:21.319605 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:21.319572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p5q6j" event={"ID":"179f2ddc-df25-46bf-b640-94826d4f434d","Type":"ContainerStarted","Data":"a5a6f1d9731b1703e31d9f433b2fca82b106e191b587d3406a764f5f2393b554"}
Sep 11 16:19:21.336696 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:21.336639    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8gctq" podStartSLOduration=73.446690598 podStartE2EDuration="1m18.33662276s" totalImagesPullingTime="4.889932162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:21.335854831 +0000 UTC m=+105.953824331" watchObservedRunningTime="2026-09-11 16:19:21.33662276 +0000 UTC m=+105.954592259"
Sep 11 16:19:21.354747 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:21.354682    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p5q6j" podStartSLOduration=73.359942387 podStartE2EDuration="1m18.354659187s" totalImagesPullingTime="4.9947168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:21.354137542 +0000 UTC m=+105.972107044" watchObservedRunningTime="2026-09-11 16:19:21.354659187 +0000 UTC m=+105.972628689"
Sep 11 16:19:37.138256 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.138212    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:37.141928 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.141910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.144784 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.144764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 16:19:37.145092 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 16:19:37.145306 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145282    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 16:19:37.145386 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145355    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 16:19:37.145446 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145410    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 16:19:37.145502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 16:19:37.145502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145461    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gkktx\""
Sep 11 16:19:37.145691 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.145678    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 16:19:37.146465 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.146449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 16:19:37.151008 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.150988    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 16:19:37.157799 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.157777    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:37.247416 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247416 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fvq2\" (UniqueName: \"kubernetes.io/projected/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-kube-api-access-6fvq2\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.247827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.247784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348685 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348685 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fvq2\" (UniqueName: \"kubernetes.io/projected/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-kube-api-access-6fvq2\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.348851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.349169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.349169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.349169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.349169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.349169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.348942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.349924 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.349894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.350026 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.349939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-config-out\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352424 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-web-config\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352508 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-config-volume\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352770 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352941 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352941 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.352941 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.353083 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.352973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.354653 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.354636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.357747 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.357725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fvq2\" (UniqueName: \"kubernetes.io/projected/dcb1c722-1f26-40d4-b9eb-75919fec6ccb-kube-api-access-6fvq2\") pod \"alertmanager-main-0\" (UID: \"dcb1c722-1f26-40d4-b9eb-75919fec6ccb\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.452070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.451959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 16:19:37.732313 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:37.732272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 16:19:37.734588 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:19:37.734564    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb1c722_1f26_40d4_b9eb_75919fec6ccb.slice/crio-b3b9553cfabd7d1160ebcf869b351060426bf393b950c74d724bed836d80c2c2 WatchSource:0}: Error finding container b3b9553cfabd7d1160ebcf869b351060426bf393b950c74d724bed836d80c2c2: Status 404 returned error can't find the container with id b3b9553cfabd7d1160ebcf869b351060426bf393b950c74d724bed836d80c2c2
Sep 11 16:19:38.376348 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:38.376293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"b3b9553cfabd7d1160ebcf869b351060426bf393b950c74d724bed836d80c2c2"}
Sep 11 16:19:39.381080 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:39.381022    2578 generic.go:356] "Generic (PLEG): container finished" podID="dcb1c722-1f26-40d4-b9eb-75919fec6ccb" containerID="404e615602a018a71137c27083880b3734551773e6ec8e1aa9c0a47802395d43" exitCode=0
Sep 11 16:19:39.381550 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:39.381090    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerDied","Data":"404e615602a018a71137c27083880b3734551773e6ec8e1aa9c0a47802395d43"}
Sep 11 16:19:41.390659 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:41.390620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"4a3be595c9626e5f088392f18849fffb06832834b153f3828d047e9b5e609094"}
Sep 11 16:19:41.391027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:41.390719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"ac2368be4203d8426f3ae0d8b976f577625a3934d168680ec428ba4dab9e78f8"}
Sep 11 16:19:41.391027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:41.390745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"c695fd5d3a6bae4a09d22fa5ea636c129479286d6da75cc6dfb09be0313c082e"}
Sep 11 16:19:41.391027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:41.390770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"da3cbc7c8da35c8cb5a0b5cb3ba6e1e9007e9d326118cb8d64a51ef09bc70b91"}
Sep 11 16:19:41.391027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:41.390792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"36c841b7aeb5359bd03ce214c9d47b728235f0fee58ee70d983c2f200b85b12c"}
Sep 11 16:19:42.398246 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:42.398211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dcb1c722-1f26-40d4-b9eb-75919fec6ccb","Type":"ContainerStarted","Data":"5c6267996d9fdcd1f83d6535080aa32ab58e4b7e06a4fcf7f1d936b54b591956"}
Sep 11 16:19:42.426085 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:42.426011    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.271999413 podStartE2EDuration="5.42599312s" totalImagesPullingTime="3.135627415s" totalInitContainerRuntime="18.366292ms" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:42.424131705 +0000 UTC m=+127.042101207" watchObservedRunningTime="2026-09-11 16:19:42.42599312 +0000 UTC m=+127.043962620"
Sep 11 16:19:52.324610 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:19:52.324578    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8gctq"
Sep 11 16:21:07.526092 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.526037    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"]
Sep 11 16:21:07.529119 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.529100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.532091 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.532068    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 16:21:07.533133 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.533113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fp5xh\""
Sep 11 16:21:07.533259 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.533117    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 16:21:07.537491 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.537465    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"]
Sep 11 16:21:07.556433 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.556393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.556620 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.556453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg62\" (UniqueName: \"kubernetes.io/projected/e3b659c1-5659-4c85-9c82-40a8b5e55e27-kube-api-access-mdg62\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.556620 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.556475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.657827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.657790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg62\" (UniqueName: \"kubernetes.io/projected/e3b659c1-5659-4c85-9c82-40a8b5e55e27-kube-api-access-mdg62\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.657984 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.657840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.657984 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.657883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.658290 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.658271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.658362 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.658300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.667953 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.667928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg62\" (UniqueName: \"kubernetes.io/projected/e3b659c1-5659-4c85-9c82-40a8b5e55e27-kube-api-access-mdg62\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:07.838572 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:07.838467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:08.113598 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:08.113561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"]
Sep 11 16:21:08.117671 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:21:08.117644    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b659c1_5659_4c85_9c82_40a8b5e55e27.slice/crio-43a6dc76f72d5f7b1c124d90fff7ce29023ab9ed89792e867e73c566d5cd243c WatchSource:0}: Error finding container 43a6dc76f72d5f7b1c124d90fff7ce29023ab9ed89792e867e73c566d5cd243c: Status 404 returned error can't find the container with id 43a6dc76f72d5f7b1c124d90fff7ce29023ab9ed89792e867e73c566d5cd243c
Sep 11 16:21:08.672365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:08.672323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" event={"ID":"e3b659c1-5659-4c85-9c82-40a8b5e55e27","Type":"ContainerStarted","Data":"43a6dc76f72d5f7b1c124d90fff7ce29023ab9ed89792e867e73c566d5cd243c"}
Sep 11 16:21:14.695786 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:14.695747    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerID="efea728ae0ce64b4a0e5a3653922b0031445018fefba0b17c46b0b62a3dfa4a2" exitCode=0
Sep 11 16:21:14.696245 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:14.695841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" event={"ID":"e3b659c1-5659-4c85-9c82-40a8b5e55e27","Type":"ContainerDied","Data":"efea728ae0ce64b4a0e5a3653922b0031445018fefba0b17c46b0b62a3dfa4a2"}
Sep 11 16:21:17.707691 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:17.707656    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerID="deab0487adaed50f2b49fb882ea938eab5905408d025b3f79c2752af1b050166" exitCode=0
Sep 11 16:21:17.707691 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:17.707695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" event={"ID":"e3b659c1-5659-4c85-9c82-40a8b5e55e27","Type":"ContainerDied","Data":"deab0487adaed50f2b49fb882ea938eab5905408d025b3f79c2752af1b050166"}
Sep 11 16:21:25.739197 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:25.739166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" event={"ID":"e3b659c1-5659-4c85-9c82-40a8b5e55e27","Type":"ContainerStarted","Data":"47d3e03cf7098a73014712eb3e0841acda135bfffc2aa48fca52e42a5e78a55a"}
Sep 11 16:21:25.762254 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:25.762197    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" podStartSLOduration=2.608426508 podStartE2EDuration="18.762176031s" totalImagesPullingTime="16.153749523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:25.76081588 +0000 UTC m=+230.378785393" watchObservedRunningTime="2026-09-11 16:21:25.762176031 +0000 UTC m=+230.380145522"
Sep 11 16:21:26.745025 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:26.744992    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerID="47d3e03cf7098a73014712eb3e0841acda135bfffc2aa48fca52e42a5e78a55a" exitCode=0
Sep 11 16:21:26.745025 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:26.745031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" event={"ID":"e3b659c1-5659-4c85-9c82-40a8b5e55e27","Type":"ContainerDied","Data":"47d3e03cf7098a73014712eb3e0841acda135bfffc2aa48fca52e42a5e78a55a"}
Sep 11 16:21:27.866240 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.866215    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:27.904507 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.904473    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-bundle\") pod \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") "
Sep 11 16:21:27.904507 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.904514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-util\") pod \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") "
Sep 11 16:21:27.904710 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.904573    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdg62\" (UniqueName: \"kubernetes.io/projected/e3b659c1-5659-4c85-9c82-40a8b5e55e27-kube-api-access-mdg62\") pod \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\" (UID: \"e3b659c1-5659-4c85-9c82-40a8b5e55e27\") "
Sep 11 16:21:27.905120 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.905092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-bundle" (OuterVolumeSpecName: "bundle") pod "e3b659c1-5659-4c85-9c82-40a8b5e55e27" (UID: "e3b659c1-5659-4c85-9c82-40a8b5e55e27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:27.906892 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.906867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b659c1-5659-4c85-9c82-40a8b5e55e27-kube-api-access-mdg62" (OuterVolumeSpecName: "kube-api-access-mdg62") pod "e3b659c1-5659-4c85-9c82-40a8b5e55e27" (UID: "e3b659c1-5659-4c85-9c82-40a8b5e55e27"). InnerVolumeSpecName "kube-api-access-mdg62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:27.908569 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:27.908540    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-util" (OuterVolumeSpecName: "util") pod "e3b659c1-5659-4c85-9c82-40a8b5e55e27" (UID: "e3b659c1-5659-4c85-9c82-40a8b5e55e27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:28.005371 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:28.005273    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdg62\" (UniqueName: \"kubernetes.io/projected/e3b659c1-5659-4c85-9c82-40a8b5e55e27-kube-api-access-mdg62\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:21:28.005371 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:28.005313    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:21:28.005556 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:28.005400    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3b659c1-5659-4c85-9c82-40a8b5e55e27-util\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:21:28.753563 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:28.753530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj" event={"ID":"e3b659c1-5659-4c85-9c82-40a8b5e55e27","Type":"ContainerDied","Data":"43a6dc76f72d5f7b1c124d90fff7ce29023ab9ed89792e867e73c566d5cd243c"}
Sep 11 16:21:28.753721 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:28.753576    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a6dc76f72d5f7b1c124d90fff7ce29023ab9ed89792e867e73c566d5cd243c"
Sep 11 16:21:28.753721 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:28.753537    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anr5rj"
Sep 11 16:21:35.551774 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.551739    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"]
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.551972    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="pull"
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.551981    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="pull"
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.551990    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="util"
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.551997    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="util"
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.552003    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="extract"
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.552008    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="extract"
Sep 11 16:21:35.552252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.552068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b659c1-5659-4c85-9c82-40a8b5e55e27" containerName="extract"
Sep 11 16:21:35.555380 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.555361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.558777 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.558747    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 11 16:21:35.558976 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.558787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 11 16:21:35.559212 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.559192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dvjl8\""
Sep 11 16:21:35.559686 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.559665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 11 16:21:35.569824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.569796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"]
Sep 11 16:21:35.658809 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.658773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c24\" (UniqueName: \"kubernetes.io/projected/c3a556d4-b66d-4dfc-8e3c-bf2c040a029c-kube-api-access-65c24\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ww6qv\" (UID: \"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.658809 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.658814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c3a556d4-b66d-4dfc-8e3c-bf2c040a029c-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ww6qv\" (UID: \"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.760085 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.760030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c24\" (UniqueName: \"kubernetes.io/projected/c3a556d4-b66d-4dfc-8e3c-bf2c040a029c-kube-api-access-65c24\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ww6qv\" (UID: \"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.760222 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.760090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c3a556d4-b66d-4dfc-8e3c-bf2c040a029c-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ww6qv\" (UID: \"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.763195 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.763176    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 11 16:21:35.770530 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.770506    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 11 16:21:35.772672 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.772649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c3a556d4-b66d-4dfc-8e3c-bf2c040a029c-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ww6qv\" (UID: \"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.781769 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.781745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 11 16:21:35.791710 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.791686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c24\" (UniqueName: \"kubernetes.io/projected/c3a556d4-b66d-4dfc-8e3c-bf2c040a029c-kube-api-access-65c24\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ww6qv\" (UID: \"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:35.870344 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.870314    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dvjl8\""
Sep 11 16:21:35.877096 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:35.877074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:36.156096 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:36.156009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"]
Sep 11 16:21:36.160841 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:21:36.160813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a556d4_b66d_4dfc_8e3c_bf2c040a029c.slice/crio-846fff9fe49f06f25ce441c4b28c05b4b1a53b443e5dc85d716c7acc5aeb21d0 WatchSource:0}: Error finding container 846fff9fe49f06f25ce441c4b28c05b4b1a53b443e5dc85d716c7acc5aeb21d0: Status 404 returned error can't find the container with id 846fff9fe49f06f25ce441c4b28c05b4b1a53b443e5dc85d716c7acc5aeb21d0
Sep 11 16:21:36.780849 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:36.780810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv" event={"ID":"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c","Type":"ContainerStarted","Data":"846fff9fe49f06f25ce441c4b28c05b4b1a53b443e5dc85d716c7acc5aeb21d0"}
Sep 11 16:21:39.749667 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.749629    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-45bpw"]
Sep 11 16:21:39.751761 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.751744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.754442 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.754412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 11 16:21:39.754442 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.754434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 11 16:21:39.754622 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.754438    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8s2gk\""
Sep 11 16:21:39.762501 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.762474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-45bpw"]
Sep 11 16:21:39.793815 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.793782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv" event={"ID":"c3a556d4-b66d-4dfc-8e3c-bf2c040a029c","Type":"ContainerStarted","Data":"7ff549739a5e416113c5cbe3482e3c5cb738c452c96e137574c1498c31b4b038"}
Sep 11 16:21:39.794038 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.794018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:21:39.813645 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.813592    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv" podStartSLOduration=2.134768686 podStartE2EDuration="4.813573644s" totalImagesPullingTime="2.678804958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:39.812347983 +0000 UTC m=+244.430317495" watchObservedRunningTime="2026-09-11 16:21:39.813573644 +0000 UTC m=+244.431543145"
Sep 11 16:21:39.892850 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.892820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/056d55bf-e04e-479b-a44b-0a76965ac5ea-cabundle0\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.893040 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.892871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-certificates\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.893040 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.892963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqh7l\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-kube-api-access-bqh7l\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.993744 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.993697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqh7l\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-kube-api-access-bqh7l\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.993936 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.993762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/056d55bf-e04e-479b-a44b-0a76965ac5ea-cabundle0\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.993936 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.993803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-certificates\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:39.993936 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:21:39.993914    2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 11 16:21:39.993936 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:21:39.993931    2578 secret.go:281] references non-existent secret key: ca.crt
Sep 11 16:21:39.994167 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:21:39.993943    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 11 16:21:39.994167 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:21:39.993957    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-45bpw: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 11 16:21:39.994167 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:21:39.994028    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-certificates podName:056d55bf-e04e-479b-a44b-0a76965ac5ea nodeName:}" failed. No retries permitted until 2026-09-11 16:21:40.494009843 +0000 UTC m=+245.111979329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-certificates") pod "keda-operator-579c8958f7-45bpw" (UID: "056d55bf-e04e-479b-a44b-0a76965ac5ea") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 11 16:21:39.994440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:39.994422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/056d55bf-e04e-479b-a44b-0a76965ac5ea-cabundle0\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:40.012900 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.012828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqh7l\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-kube-api-access-bqh7l\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:40.084706 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.084667    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"]
Sep 11 16:21:40.086891 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.086872    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.089531 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.089512    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 11 16:21:40.099637 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.099609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"]
Sep 11 16:21:40.195106 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.195038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748gt\" (UniqueName: \"kubernetes.io/projected/6c588022-e635-4f43-a801-4bfdf6335415-kube-api-access-748gt\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.195260 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.195157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6c588022-e635-4f43-a801-4bfdf6335415-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.195260 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.195186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c588022-e635-4f43-a801-4bfdf6335415-certificates\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.295756 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.295658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6c588022-e635-4f43-a801-4bfdf6335415-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.295756 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.295711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c588022-e635-4f43-a801-4bfdf6335415-certificates\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.295977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.295770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748gt\" (UniqueName: \"kubernetes.io/projected/6c588022-e635-4f43-a801-4bfdf6335415-kube-api-access-748gt\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.296129 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.296108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/6c588022-e635-4f43-a801-4bfdf6335415-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.298547 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.298527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/6c588022-e635-4f43-a801-4bfdf6335415-certificates\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.305792 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.305768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748gt\" (UniqueName: \"kubernetes.io/projected/6c588022-e635-4f43-a801-4bfdf6335415-kube-api-access-748gt\") pod \"keda-metrics-apiserver-77857b7b98-4tsf9\" (UID: \"6c588022-e635-4f43-a801-4bfdf6335415\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.397070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.397005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:40.430793 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.430762    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-sbk8w"]
Sep 11 16:21:40.433099 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.433075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.435725 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.435527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 11 16:21:40.444870 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.444826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-sbk8w"]
Sep 11 16:21:40.498226 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.498187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab850bbd-69ab-4e2f-b081-8c59c99e9e7d-certificates\") pod \"keda-admission-6c4d864b86-sbk8w\" (UID: \"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d\") " pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.498226 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.498230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj68n\" (UniqueName: \"kubernetes.io/projected/ab850bbd-69ab-4e2f-b081-8c59c99e9e7d-kube-api-access-mj68n\") pod \"keda-admission-6c4d864b86-sbk8w\" (UID: \"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d\") " pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.498467 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.498259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-certificates\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:40.500773 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.500753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/056d55bf-e04e-479b-a44b-0a76965ac5ea-certificates\") pod \"keda-operator-579c8958f7-45bpw\" (UID: \"056d55bf-e04e-479b-a44b-0a76965ac5ea\") " pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:40.599153 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.599039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab850bbd-69ab-4e2f-b081-8c59c99e9e7d-certificates\") pod \"keda-admission-6c4d864b86-sbk8w\" (UID: \"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d\") " pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.599153 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.599114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj68n\" (UniqueName: \"kubernetes.io/projected/ab850bbd-69ab-4e2f-b081-8c59c99e9e7d-kube-api-access-mj68n\") pod \"keda-admission-6c4d864b86-sbk8w\" (UID: \"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d\") " pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.601698 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.601677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ab850bbd-69ab-4e2f-b081-8c59c99e9e7d-certificates\") pod \"keda-admission-6c4d864b86-sbk8w\" (UID: \"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d\") " pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.608635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.608610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj68n\" (UniqueName: \"kubernetes.io/projected/ab850bbd-69ab-4e2f-b081-8c59c99e9e7d-kube-api-access-mj68n\") pod \"keda-admission-6c4d864b86-sbk8w\" (UID: \"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d\") " pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.661630 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.661590    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:40.672715 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.672691    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"]
Sep 11 16:21:40.675118 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:21:40.675090    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c588022_e635_4f43_a801_4bfdf6335415.slice/crio-bc33619512f8ccc1a6d5bdb9c0e2a84c38f59e513a71eabfdaf8b0507fed5cd8 WatchSource:0}: Error finding container bc33619512f8ccc1a6d5bdb9c0e2a84c38f59e513a71eabfdaf8b0507fed5cd8: Status 404 returned error can't find the container with id bc33619512f8ccc1a6d5bdb9c0e2a84c38f59e513a71eabfdaf8b0507fed5cd8
Sep 11 16:21:40.746097 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.746034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:40.800252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.800215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9" event={"ID":"6c588022-e635-4f43-a801-4bfdf6335415","Type":"ContainerStarted","Data":"bc33619512f8ccc1a6d5bdb9c0e2a84c38f59e513a71eabfdaf8b0507fed5cd8"}
Sep 11 16:21:40.935200 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:40.935166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-45bpw"]
Sep 11 16:21:40.938319 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:21:40.938289    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056d55bf_e04e_479b_a44b_0a76965ac5ea.slice/crio-a6d8fdb741ea27e6375d781d9f4ed0c5f4fe593d210d2eef30fec5dd3707b75e WatchSource:0}: Error finding container a6d8fdb741ea27e6375d781d9f4ed0c5f4fe593d210d2eef30fec5dd3707b75e: Status 404 returned error can't find the container with id a6d8fdb741ea27e6375d781d9f4ed0c5f4fe593d210d2eef30fec5dd3707b75e
Sep 11 16:21:41.014913 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:41.014882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-sbk8w"]
Sep 11 16:21:41.018073 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:21:41.018033    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab850bbd_69ab_4e2f_b081_8c59c99e9e7d.slice/crio-eb22b25b16b3819a4087ad4363c3e85e9957024e522ee2fb48f6b27306062fcb WatchSource:0}: Error finding container eb22b25b16b3819a4087ad4363c3e85e9957024e522ee2fb48f6b27306062fcb: Status 404 returned error can't find the container with id eb22b25b16b3819a4087ad4363c3e85e9957024e522ee2fb48f6b27306062fcb
Sep 11 16:21:41.808282 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:41.808236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-45bpw" event={"ID":"056d55bf-e04e-479b-a44b-0a76965ac5ea","Type":"ContainerStarted","Data":"a6d8fdb741ea27e6375d781d9f4ed0c5f4fe593d210d2eef30fec5dd3707b75e"}
Sep 11 16:21:41.812256 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:41.811974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w" event={"ID":"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d","Type":"ContainerStarted","Data":"eb22b25b16b3819a4087ad4363c3e85e9957024e522ee2fb48f6b27306062fcb"}
Sep 11 16:21:44.826272 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.826240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-45bpw" event={"ID":"056d55bf-e04e-479b-a44b-0a76965ac5ea","Type":"ContainerStarted","Data":"a65a826b4e5632d1eb16a6f604a4a567f7139aec917bab2197bba3f0d4f2954e"}
Sep 11 16:21:44.826705 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.826307    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:21:44.827863 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.827837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9" event={"ID":"6c588022-e635-4f43-a801-4bfdf6335415","Type":"ContainerStarted","Data":"3a6ccee8d8b781c3e42ed315ef40e37a56ab2b0bae5ddcc64a521e0c7484edaa"}
Sep 11 16:21:44.827988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.827940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:21:44.829379 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.829359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w" event={"ID":"ab850bbd-69ab-4e2f-b081-8c59c99e9e7d","Type":"ContainerStarted","Data":"77e29ca11f9e071466b79184ac969590fbc3563deb0b7a4bb8de5c4338e3891b"}
Sep 11 16:21:44.829494 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.829482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:21:44.844763 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.844717    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-45bpw" podStartSLOduration=2.140727669 podStartE2EDuration="5.844702598s" totalImagesPullingTime="3.703974929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:44.84359747 +0000 UTC m=+249.461566971" watchObservedRunningTime="2026-09-11 16:21:44.844702598 +0000 UTC m=+249.462672097"
Sep 11 16:21:44.868349 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.868256    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9" podStartSLOduration=0.905112473 podStartE2EDuration="4.868239844s" totalImagesPullingTime="3.963127371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:44.866533798 +0000 UTC m=+249.484503298" watchObservedRunningTime="2026-09-11 16:21:44.868239844 +0000 UTC m=+249.486209395"
Sep 11 16:21:44.885860 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:44.885813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w" podStartSLOduration=1.271386793 podStartE2EDuration="4.885798234s" totalImagesPullingTime="3.614411441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:44.885476924 +0000 UTC m=+249.503446426" watchObservedRunningTime="2026-09-11 16:21:44.885798234 +0000 UTC m=+249.503767734"
Sep 11 16:21:55.838070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:21:55.838026    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-4tsf9"
Sep 11 16:22:00.802784 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:22:00.802746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ww6qv"
Sep 11 16:22:05.835632 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:22:05.835600    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-sbk8w"
Sep 11 16:22:05.836077 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:22:05.835684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-45bpw"
Sep 11 16:22:35.780393 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:22:35.780367    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:23:00.627186 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.627153    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-6nxxw"]
Sep 11 16:23:00.629502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.629485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:00.632548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.632494    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 11 16:23:00.632694 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.632641    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-rd56s\""
Sep 11 16:23:00.632694 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.632650    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:23:00.633754 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.633725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:23:00.636396 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.636070    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"]
Sep 11 16:23:00.638342 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.638329    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:00.640953 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.640939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 11 16:23:00.641083 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.641061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-f2fvh\""
Sep 11 16:23:00.647240 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.647215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-6nxxw"]
Sep 11 16:23:00.655134 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.655105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"]
Sep 11 16:23:00.670377 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.670345    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-rfgbr"]
Sep 11 16:23:00.672589 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.672565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.678680 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.678659    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 11 16:23:00.678781 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.678683    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9m8qd\""
Sep 11 16:23:00.683005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.682974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5zk\" (UniqueName: \"kubernetes.io/projected/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-kube-api-access-bn5zk\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:00.683005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.682995    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rfgbr"]
Sep 11 16:23:00.683182 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.683031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrtds\" (UniqueName: \"kubernetes.io/projected/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-kube-api-access-qrtds\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:00.683219 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.683178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:00.683219 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.683202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:00.784570 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.784534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5zk\" (UniqueName: \"kubernetes.io/projected/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-kube-api-access-bn5zk\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:00.784570 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.784574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44d6fc10-93fc-4a7a-abcc-d620ea89c42f-data\") pod \"seaweedfs-64568bcd49-rfgbr\" (UID: \"44d6fc10-93fc-4a7a-abcc-d620ea89c42f\") " pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.784778 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.784619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrtds\" (UniqueName: \"kubernetes.io/projected/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-kube-api-access-qrtds\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:00.784814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.784789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smnc\" (UniqueName: \"kubernetes.io/projected/44d6fc10-93fc-4a7a-abcc-d620ea89c42f-kube-api-access-8smnc\") pod \"seaweedfs-64568bcd49-rfgbr\" (UID: \"44d6fc10-93fc-4a7a-abcc-d620ea89c42f\") " pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.784851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.784842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:00.784891 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.784877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:00.784947 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:23:00.784932    2578 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 16:23:00.784982 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:23:00.784960    2578 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 11 16:23:00.785014 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:23:00.785006    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert podName:2e77a30f-25a2-4931-ad8e-5a8b63cf54c8 nodeName:}" failed. No retries permitted until 2026-09-11 16:23:01.284990694 +0000 UTC m=+325.902960172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert") pod "kserve-controller-manager-656b75fc8d-6nxxw" (UID: "2e77a30f-25a2-4931-ad8e-5a8b63cf54c8") : secret "kserve-webhook-server-cert" not found
Sep 11 16:23:00.785093 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:23:00.785020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-cert podName:7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5 nodeName:}" failed. No retries permitted until 2026-09-11 16:23:01.285013702 +0000 UTC m=+325.902983181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-cert") pod "llmisvc-controller-manager-645f9c8fd6-lbx8v" (UID: "7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5") : secret "llmisvc-webhook-server-cert" not found
Sep 11 16:23:00.795826 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.795792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrtds\" (UniqueName: \"kubernetes.io/projected/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-kube-api-access-qrtds\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:00.795960 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.795829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5zk\" (UniqueName: \"kubernetes.io/projected/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-kube-api-access-bn5zk\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:00.886173 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.886080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smnc\" (UniqueName: \"kubernetes.io/projected/44d6fc10-93fc-4a7a-abcc-d620ea89c42f-kube-api-access-8smnc\") pod \"seaweedfs-64568bcd49-rfgbr\" (UID: \"44d6fc10-93fc-4a7a-abcc-d620ea89c42f\") " pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.886345 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.886196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44d6fc10-93fc-4a7a-abcc-d620ea89c42f-data\") pod \"seaweedfs-64568bcd49-rfgbr\" (UID: \"44d6fc10-93fc-4a7a-abcc-d620ea89c42f\") " pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.886548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.886531    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/44d6fc10-93fc-4a7a-abcc-d620ea89c42f-data\") pod \"seaweedfs-64568bcd49-rfgbr\" (UID: \"44d6fc10-93fc-4a7a-abcc-d620ea89c42f\") " pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.900753 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.900724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smnc\" (UniqueName: \"kubernetes.io/projected/44d6fc10-93fc-4a7a-abcc-d620ea89c42f-kube-api-access-8smnc\") pod \"seaweedfs-64568bcd49-rfgbr\" (UID: \"44d6fc10-93fc-4a7a-abcc-d620ea89c42f\") " pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:00.981285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:00.981247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:01.258451 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.258367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-rfgbr"]
Sep 11 16:23:01.261255 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:23:01.261229    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d6fc10_93fc_4a7a_abcc_d620ea89c42f.slice/crio-233fc15ba4acd7faff14e6dfa81f77cd014421dcfe885877b1a4e43788459548 WatchSource:0}: Error finding container 233fc15ba4acd7faff14e6dfa81f77cd014421dcfe885877b1a4e43788459548: Status 404 returned error can't find the container with id 233fc15ba4acd7faff14e6dfa81f77cd014421dcfe885877b1a4e43788459548
Sep 11 16:23:01.262595 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.262578    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:23:01.289501 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.289463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:01.289711 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.289510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:01.292168 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.292142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-lbx8v\" (UID: \"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:01.292282 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.292173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert\") pod \"kserve-controller-manager-656b75fc8d-6nxxw\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") " pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:01.539968 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.539880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:01.547887 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.547857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:01.814977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.814949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-6nxxw"]
Sep 11 16:23:01.816886 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:23:01.816858    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e77a30f_25a2_4931_ad8e_5a8b63cf54c8.slice/crio-9a850611cc7f072cc709c0e81e20c9180c8b176187be3059355949765999110b WatchSource:0}: Error finding container 9a850611cc7f072cc709c0e81e20c9180c8b176187be3059355949765999110b: Status 404 returned error can't find the container with id 9a850611cc7f072cc709c0e81e20c9180c8b176187be3059355949765999110b
Sep 11 16:23:01.826999 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:01.826957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"]
Sep 11 16:23:01.829638 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:23:01.829612    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a3e6100_5b0b_4685_b5c3_ae9ebb82e1c5.slice/crio-2f1362629fbbb0662f9b52e1d425c7fccd7be76ecd40f7cdf76099b9ec4f40cb WatchSource:0}: Error finding container 2f1362629fbbb0662f9b52e1d425c7fccd7be76ecd40f7cdf76099b9ec4f40cb: Status 404 returned error can't find the container with id 2f1362629fbbb0662f9b52e1d425c7fccd7be76ecd40f7cdf76099b9ec4f40cb
Sep 11 16:23:02.110560 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:02.110507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v" event={"ID":"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5","Type":"ContainerStarted","Data":"2f1362629fbbb0662f9b52e1d425c7fccd7be76ecd40f7cdf76099b9ec4f40cb"}
Sep 11 16:23:02.111846 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:02.111812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rfgbr" event={"ID":"44d6fc10-93fc-4a7a-abcc-d620ea89c42f","Type":"ContainerStarted","Data":"233fc15ba4acd7faff14e6dfa81f77cd014421dcfe885877b1a4e43788459548"}
Sep 11 16:23:02.113192 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:02.113164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw" event={"ID":"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8","Type":"ContainerStarted","Data":"9a850611cc7f072cc709c0e81e20c9180c8b176187be3059355949765999110b"}
Sep 11 16:23:07.140152 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.140114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v" event={"ID":"7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5","Type":"ContainerStarted","Data":"82bea47c7037806966eb47ff55bd8feca3aff25e6f3fff0e4d5bd624a6e3bf45"}
Sep 11 16:23:07.140678 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.140218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:07.141760 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.141739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-rfgbr" event={"ID":"44d6fc10-93fc-4a7a-abcc-d620ea89c42f","Type":"ContainerStarted","Data":"8c2d057238953df8bfcb5596fced27515bbc02caa463e41dc8d207666b3ba0ca"}
Sep 11 16:23:07.141870 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.141832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:07.143236 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.143216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw" event={"ID":"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8","Type":"ContainerStarted","Data":"0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67"}
Sep 11 16:23:07.143366 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.143327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:07.156738 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.156677    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v" podStartSLOduration=2.655172758 podStartE2EDuration="7.156656072s" totalImagesPullingTime="4.501483314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:07.156511128 +0000 UTC m=+331.774480629" watchObservedRunningTime="2026-09-11 16:23:07.156656072 +0000 UTC m=+331.774625576"
Sep 11 16:23:07.177477 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.177422    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-rfgbr" podStartSLOduration=2.070778717 podStartE2EDuration="7.177406017s" totalImagesPullingTime="5.1066273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:07.17701645 +0000 UTC m=+331.794985952" watchObservedRunningTime="2026-09-11 16:23:07.177406017 +0000 UTC m=+331.795375523"
Sep 11 16:23:07.194726 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:07.194674    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw" podStartSLOduration=2.752440737 podStartE2EDuration="7.194656438s" totalImagesPullingTime="4.442215701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:07.193070188 +0000 UTC m=+331.811039684" watchObservedRunningTime="2026-09-11 16:23:07.194656438 +0000 UTC m=+331.812625963"
Sep 11 16:23:13.150185 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:13.150155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-rfgbr"
Sep 11 16:23:38.149620 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:38.149529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-lbx8v"
Sep 11 16:23:38.152602 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:38.152579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:47.788719 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.788681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-6nxxw"]
Sep 11 16:23:47.789126 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.788935    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw" podUID="2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" containerName="manager" containerID="cri-o://0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67" gracePeriod=10
Sep 11 16:23:47.816197 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.816167    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-b6qrd"]
Sep 11 16:23:47.818200 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.818185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:47.829581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.829553    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-b6qrd"]
Sep 11 16:23:47.959965 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.959909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d7bbac-98e2-447d-aa8e-9097bdaee58d-cert\") pod \"kserve-controller-manager-656b75fc8d-b6qrd\" (UID: \"57d7bbac-98e2-447d-aa8e-9097bdaee58d\") " pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:47.960267 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:47.960221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcr6s\" (UniqueName: \"kubernetes.io/projected/57d7bbac-98e2-447d-aa8e-9097bdaee58d-kube-api-access-kcr6s\") pod \"kserve-controller-manager-656b75fc8d-b6qrd\" (UID: \"57d7bbac-98e2-447d-aa8e-9097bdaee58d\") " pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:48.040913 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.040849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:48.060648 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.060613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert\") pod \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") "
Sep 11 16:23:48.060915 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.060898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrtds\" (UniqueName: \"kubernetes.io/projected/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-kube-api-access-qrtds\") pod \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\" (UID: \"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8\") "
Sep 11 16:23:48.061126 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.061107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d7bbac-98e2-447d-aa8e-9097bdaee58d-cert\") pod \"kserve-controller-manager-656b75fc8d-b6qrd\" (UID: \"57d7bbac-98e2-447d-aa8e-9097bdaee58d\") " pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:48.061330 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.061315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcr6s\" (UniqueName: \"kubernetes.io/projected/57d7bbac-98e2-447d-aa8e-9097bdaee58d-kube-api-access-kcr6s\") pod \"kserve-controller-manager-656b75fc8d-b6qrd\" (UID: \"57d7bbac-98e2-447d-aa8e-9097bdaee58d\") " pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:48.064301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.064274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert" (OuterVolumeSpecName: "cert") pod "2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" (UID: "2e77a30f-25a2-4931-ad8e-5a8b63cf54c8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:23:48.064535 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.064506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-kube-api-access-qrtds" (OuterVolumeSpecName: "kube-api-access-qrtds") pod "2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" (UID: "2e77a30f-25a2-4931-ad8e-5a8b63cf54c8"). InnerVolumeSpecName "kube-api-access-qrtds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:23:48.068511 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.068458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57d7bbac-98e2-447d-aa8e-9097bdaee58d-cert\") pod \"kserve-controller-manager-656b75fc8d-b6qrd\" (UID: \"57d7bbac-98e2-447d-aa8e-9097bdaee58d\") " pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:48.072789 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.072760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcr6s\" (UniqueName: \"kubernetes.io/projected/57d7bbac-98e2-447d-aa8e-9097bdaee58d-kube-api-access-kcr6s\") pod \"kserve-controller-manager-656b75fc8d-b6qrd\" (UID: \"57d7bbac-98e2-447d-aa8e-9097bdaee58d\") " pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:48.162301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.162264    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrtds\" (UniqueName: \"kubernetes.io/projected/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-kube-api-access-qrtds\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:23:48.162301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.162293    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:23:48.174251 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.174222    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:48.292702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.292615    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" containerID="0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67" exitCode=0
Sep 11 16:23:48.292702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.292664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw"
Sep 11 16:23:48.292872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.292706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw" event={"ID":"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8","Type":"ContainerDied","Data":"0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67"}
Sep 11 16:23:48.292872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.292791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-6nxxw" event={"ID":"2e77a30f-25a2-4931-ad8e-5a8b63cf54c8","Type":"ContainerDied","Data":"9a850611cc7f072cc709c0e81e20c9180c8b176187be3059355949765999110b"}
Sep 11 16:23:48.292872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.292818    2578 scope.go:117] "RemoveContainer" containerID="0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67"
Sep 11 16:23:48.301888 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.301866    2578 scope.go:117] "RemoveContainer" containerID="0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67"
Sep 11 16:23:48.302240 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:23:48.302217    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67\": container with ID starting with 0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67 not found: ID does not exist" containerID="0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67"
Sep 11 16:23:48.302318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.302254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67"} err="failed to get container status \"0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67\": rpc error: code = NotFound desc = could not find container \"0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67\": container with ID starting with 0737c7eb2095801dcb446a14686838109f19acc0b8dceca52462a8bcdea44a67 not found: ID does not exist"
Sep 11 16:23:48.316561 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.316531    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-6nxxw"]
Sep 11 16:23:48.320321 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.320296    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-6nxxw"]
Sep 11 16:23:48.444659 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:48.444628    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-b6qrd"]
Sep 11 16:23:48.447720 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:23:48.447689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d7bbac_98e2_447d_aa8e_9097bdaee58d.slice/crio-4dd713664eb7461416232c02d0dbfafc3ef3d0e8d62486fc498199b0a1b27518 WatchSource:0}: Error finding container 4dd713664eb7461416232c02d0dbfafc3ef3d0e8d62486fc498199b0a1b27518: Status 404 returned error can't find the container with id 4dd713664eb7461416232c02d0dbfafc3ef3d0e8d62486fc498199b0a1b27518
Sep 11 16:23:49.297903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:49.297866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd" event={"ID":"57d7bbac-98e2-447d-aa8e-9097bdaee58d","Type":"ContainerStarted","Data":"a2c0a35d671b63b69a2acff0734a8d16da1fe54e323c1af0e671e572336070ab"}
Sep 11 16:23:49.298393 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:49.297924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd" event={"ID":"57d7bbac-98e2-447d-aa8e-9097bdaee58d","Type":"ContainerStarted","Data":"4dd713664eb7461416232c02d0dbfafc3ef3d0e8d62486fc498199b0a1b27518"}
Sep 11 16:23:49.298393 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:49.297959    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:23:49.316201 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:49.316141    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd" podStartSLOduration=1.929627145 podStartE2EDuration="2.316122634s" totalImagesPullingTime="386.495489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:49.314728732 +0000 UTC m=+373.932698233" watchObservedRunningTime="2026-09-11 16:23:49.316122634 +0000 UTC m=+373.934092147"
Sep 11 16:23:49.895630 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:23:49.895593    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" path="/var/lib/kubelet/pods/2e77a30f-25a2-4931-ad8e-5a8b63cf54c8/volumes"
Sep 11 16:24:20.307961 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:20.307932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-656b75fc8d-b6qrd"
Sep 11 16:24:21.583374 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.583341    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-7l97w"]
Sep 11 16:24:21.583824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.583798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" containerName="manager"
Sep 11 16:24:21.583824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.583817    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" containerName="manager"
Sep 11 16:24:21.583933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.583891    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e77a30f-25a2-4931-ad8e-5a8b63cf54c8" containerName="manager"
Sep 11 16:24:21.586906 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.586886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:21.589675 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.589652    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 11 16:24:21.589675 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.589673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qpvzb\""
Sep 11 16:24:21.596864 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.596838    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7l97w"]
Sep 11 16:24:21.599228 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.599205    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-tkhrk"]
Sep 11 16:24:21.602732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.602667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.605381 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.605354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 11 16:24:21.605381 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.605370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-7lnsl\""
Sep 11 16:24:21.605531 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.605418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 11 16:24:21.612882 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.612857    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-tkhrk"]
Sep 11 16:24:21.702471 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.702428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkgl\" (UniqueName: \"kubernetes.io/projected/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-kube-api-access-mfkgl\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.702662 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.702486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.702662 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.702578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-cert\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.702662 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.702606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-tls-certs\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:21.702662 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.702625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5b69\" (UniqueName: \"kubernetes.io/projected/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-kube-api-access-r5b69\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:21.803985 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.803952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-cert\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.803985 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.803988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-tls-certs\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:21.804229 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.804008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5b69\" (UniqueName: \"kubernetes.io/projected/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-kube-api-access-r5b69\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:21.804229 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.804073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkgl\" (UniqueName: \"kubernetes.io/projected/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-kube-api-access-mfkgl\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.804229 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.804107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.804229 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:24:21.804151    2578 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 16:24:21.804229 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:24:21.804193    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 11 16:24:21.804229 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:24:21.804233    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-tls-certs podName:fe97ef8c-b207-4410-ad9d-a3e1a9cc1879 nodeName:}" failed. No retries permitted until 2026-09-11 16:24:22.304211531 +0000 UTC m=+406.922181028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-tls-certs") pod "model-serving-api-7b954ffc95-7l97w" (UID: "fe97ef8c-b207-4410-ad9d-a3e1a9cc1879") : secret "model-serving-api-tls" not found
Sep 11 16:24:21.804468 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:24:21.804248    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-metrics-certs podName:c6033440-bb82-4e8a-bf63-e7e2d61d09d3 nodeName:}" failed. No retries permitted until 2026-09-11 16:24:22.304242237 +0000 UTC m=+406.922211716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-metrics-certs") pod "odh-model-controller-7b966d7984-tkhrk" (UID: "c6033440-bb82-4e8a-bf63-e7e2d61d09d3") : secret "odh-model-controller-metrics-tls" not found
Sep 11 16:24:21.806674 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.806650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-cert\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.812989 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.812952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkgl\" (UniqueName: \"kubernetes.io/projected/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-kube-api-access-mfkgl\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:21.813246 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:21.813223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5b69\" (UniqueName: \"kubernetes.io/projected/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-kube-api-access-r5b69\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:22.307012 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.306961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:22.307192 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.307037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-tls-certs\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:22.309589 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.309570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6033440-bb82-4e8a-bf63-e7e2d61d09d3-metrics-certs\") pod \"odh-model-controller-7b966d7984-tkhrk\" (UID: \"c6033440-bb82-4e8a-bf63-e7e2d61d09d3\") " pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:22.309715 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.309694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe97ef8c-b207-4410-ad9d-a3e1a9cc1879-tls-certs\") pod \"model-serving-api-7b954ffc95-7l97w\" (UID: \"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879\") " pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:22.497130 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.497088    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:22.513687 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.513656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:22.772473 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.772434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7l97w"]
Sep 11 16:24:22.776307 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:24:22.776264    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe97ef8c_b207_4410_ad9d_a3e1a9cc1879.slice/crio-5d2afd1d6c7fe1b1fac0e217de249c4afd91342713e409f5a85d759962047286 WatchSource:0}: Error finding container 5d2afd1d6c7fe1b1fac0e217de249c4afd91342713e409f5a85d759962047286: Status 404 returned error can't find the container with id 5d2afd1d6c7fe1b1fac0e217de249c4afd91342713e409f5a85d759962047286
Sep 11 16:24:22.795096 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:22.795040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-tkhrk"]
Sep 11 16:24:22.797683 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:24:22.797657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6033440_bb82_4e8a_bf63_e7e2d61d09d3.slice/crio-80c579ec2837d4dfdb0a7ce5a02e262cd7ee5478e8eb0c84564987e0881c6b07 WatchSource:0}: Error finding container 80c579ec2837d4dfdb0a7ce5a02e262cd7ee5478e8eb0c84564987e0881c6b07: Status 404 returned error can't find the container with id 80c579ec2837d4dfdb0a7ce5a02e262cd7ee5478e8eb0c84564987e0881c6b07
Sep 11 16:24:23.429261 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:23.429218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-tkhrk" event={"ID":"c6033440-bb82-4e8a-bf63-e7e2d61d09d3","Type":"ContainerStarted","Data":"80c579ec2837d4dfdb0a7ce5a02e262cd7ee5478e8eb0c84564987e0881c6b07"}
Sep 11 16:24:23.430613 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:23.430581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7l97w" event={"ID":"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879","Type":"ContainerStarted","Data":"5d2afd1d6c7fe1b1fac0e217de249c4afd91342713e409f5a85d759962047286"}
Sep 11 16:24:26.447433 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:26.447391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7l97w" event={"ID":"fe97ef8c-b207-4410-ad9d-a3e1a9cc1879","Type":"ContainerStarted","Data":"70aef9dba8dbad3314e543951e059b1f2cb40512324d2003aa96757da9438c3e"}
Sep 11 16:24:26.447887 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:26.447454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:24:26.448885 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:26.448860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-tkhrk" event={"ID":"c6033440-bb82-4e8a-bf63-e7e2d61d09d3","Type":"ContainerStarted","Data":"ab9de1c5f2ebc4b65fc5e1e4b964de01254f52f481a4018c2d1c8075700bd8b4"}
Sep 11 16:24:26.448981 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:26.448964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:26.465801 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:26.465756    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-7l97w" podStartSLOduration=2.613342675 podStartE2EDuration="5.465738706s" totalImagesPullingTime="2.852396031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:26.465570167 +0000 UTC m=+411.083539668" watchObservedRunningTime="2026-09-11 16:24:26.465738706 +0000 UTC m=+411.083708208"
Sep 11 16:24:26.481981 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:26.481920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-tkhrk" podStartSLOduration=2.591350167 podStartE2EDuration="5.481898028s" totalImagesPullingTime="2.890547861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:26.480181744 +0000 UTC m=+411.098151246" watchObservedRunningTime="2026-09-11 16:24:26.481898028 +0000 UTC m=+411.099867530"
Sep 11 16:24:37.456909 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:37.456881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-tkhrk"
Sep 11 16:24:37.459073 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:24:37.459032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-7l97w"
Sep 11 16:26:09.925154 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.925117    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"]
Sep 11 16:26:09.928911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.928887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:09.931742 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.931702    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:26:09.931930 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.931911    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a0b0b-kube-rbac-proxy-sar-config\""
Sep 11 16:26:09.932011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.931917    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:26:09.932011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.931945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:26:09.932284 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.932268    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:26:09.932354 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.932338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a0b0b-predictor-serving-cert\""
Sep 11 16:26:09.932868 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.932855    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:26:09.938839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:09.938811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"]
Sep 11 16:26:10.028379 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.028345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.028566 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.028400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.028566 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.028465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.028566 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.028551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrnk\" (UniqueName: \"kubernetes.io/projected/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-api-access-tnrnk\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.028708 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.028592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb518af5-500c-43d3-a3c8-5e81dcaf0977-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.129198 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.129167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrnk\" (UniqueName: \"kubernetes.io/projected/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-api-access-tnrnk\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.129441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.129216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb518af5-500c-43d3-a3c8-5e81dcaf0977-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.129441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.129244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.129441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.129275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.129441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.129328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.130069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.129997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.130069 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.130036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.130353 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.130145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.131902 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.131880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb518af5-500c-43d3-a3c8-5e81dcaf0977-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.138650 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.138619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrnk\" (UniqueName: \"kubernetes.io/projected/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-api-access-tnrnk\") pod \"isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.240584 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.240510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:10.307195 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.307157    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"]
Sep 11 16:26:10.311861 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.311832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.314795 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.314769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 11 16:26:10.314922 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.314802    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 11 16:26:10.320919 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.320825    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"]
Sep 11 16:26:10.431783 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.431752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.431933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.431802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b8cd84-47ec-4163-8508-bd889ffe196c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.431933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.431889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.431933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.431909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.431933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.431926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzkw\" (UniqueName: \"kubernetes.io/projected/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-api-access-2kzkw\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.517587 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.517557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"]
Sep 11 16:26:10.519747 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:26:10.519705    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb518af5_500c_43d3_a3c8_5e81dcaf0977.slice/crio-d81c7d381375225acc100a91c2d9e4f840b44db0345714dfe27b97cc94af1aed WatchSource:0}: Error finding container d81c7d381375225acc100a91c2d9e4f840b44db0345714dfe27b97cc94af1aed: Status 404 returned error can't find the container with id d81c7d381375225acc100a91c2d9e4f840b44db0345714dfe27b97cc94af1aed
Sep 11 16:26:10.532561 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.532537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.532688 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.532569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.532688 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.532588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzkw\" (UniqueName: \"kubernetes.io/projected/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-api-access-2kzkw\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.532688 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.532613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.532688 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.532640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b8cd84-47ec-4163-8508-bd889ffe196c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.532834 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:26:10.532733    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:26:10.532834 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:26:10.532801    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls podName:81b8cd84-47ec-4163-8508-bd889ffe196c nodeName:}" failed. No retries permitted until 2026-09-11 16:26:11.032780654 +0000 UTC m=+515.650750147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" (UID: "81b8cd84-47ec-4163-8508-bd889ffe196c") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 11 16:26:10.533064 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.533029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b8cd84-47ec-4163-8508-bd889ffe196c-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.533368 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.533346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.533410 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.533354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.543134 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.543107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzkw\" (UniqueName: \"kubernetes.io/projected/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-api-access-2kzkw\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:10.848287 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:10.848212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerStarted","Data":"d81c7d381375225acc100a91c2d9e4f840b44db0345714dfe27b97cc94af1aed"}
Sep 11 16:26:11.037314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:11.037277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:11.039841 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:11.039821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-snrpj\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:11.226755 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:11.226713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:11.503764 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:11.503739    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"]
Sep 11 16:26:11.505936 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:26:11.505911    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b8cd84_47ec_4163_8508_bd889ffe196c.slice/crio-03f0a5e7ee5323191784a01b0add69378b5d5cab1bad079a7387ce42ba881450 WatchSource:0}: Error finding container 03f0a5e7ee5323191784a01b0add69378b5d5cab1bad079a7387ce42ba881450: Status 404 returned error can't find the container with id 03f0a5e7ee5323191784a01b0add69378b5d5cab1bad079a7387ce42ba881450
Sep 11 16:26:11.853603 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:11.853522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerStarted","Data":"03f0a5e7ee5323191784a01b0add69378b5d5cab1bad079a7387ce42ba881450"}
Sep 11 16:26:15.875621 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:15.875577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerStarted","Data":"36d3dbe69d2821f3f12cba2dfb9e043cf38b3884a3bae34c5e16c2cb27e2dd13"}
Sep 11 16:26:15.877211 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:15.877182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerStarted","Data":"f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5"}
Sep 11 16:26:19.896889 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:19.896848    2578 generic.go:356] "Generic (PLEG): container finished" podID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerID="36d3dbe69d2821f3f12cba2dfb9e043cf38b3884a3bae34c5e16c2cb27e2dd13" exitCode=0
Sep 11 16:26:19.897342 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:19.896910    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerDied","Data":"36d3dbe69d2821f3f12cba2dfb9e043cf38b3884a3bae34c5e16c2cb27e2dd13"}
Sep 11 16:26:19.898768 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:19.898748    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerID="f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5" exitCode=0
Sep 11 16:26:19.898832 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:19.898808    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerDied","Data":"f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5"}
Sep 11 16:26:37.026138 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:37.026040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerStarted","Data":"56ebe7eb2679486d380952eda9b3f0bb2b2c97e3a636edace2b52768f9060508"}
Sep 11 16:26:37.029720 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:37.029684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerStarted","Data":"e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2"}
Sep 11 16:26:40.045142 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:40.045105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerStarted","Data":"553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09"}
Sep 11 16:26:40.047008 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:40.046983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerStarted","Data":"43a0f234d8a747b3e51752fc9d90d5d77a09dcfb617287104f9f4220479f8b69"}
Sep 11 16:26:40.047163 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:40.047147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:40.080944 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:40.080887    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podStartSLOduration=2.847305781 podStartE2EDuration="30.080868733s" totalImagesPullingTime="23.288789203s" totalInitContainerRuntime="3.944773749s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:40.078409986 +0000 UTC m=+544.696379490" watchObservedRunningTime="2026-09-11 16:26:40.080868733 +0000 UTC m=+544.698838234"
Sep 11 16:26:41.052507 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:41.052447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:41.054062 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:41.054001    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:26:42.056271 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:42.056234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:26:43.064083 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.064030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerStarted","Data":"ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1"}
Sep 11 16:26:43.064476 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.064248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:43.064476 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.064322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:43.064476 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.064357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:43.065738 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.065698    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:26:43.065838 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.065730    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="Get \"http://10.134.0.21:9081/\": dial tcp 10.134.0.21:9081: connect: connection refused"
Sep 11 16:26:43.086979 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:43.086904    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podStartSLOduration=2.934862902 podStartE2EDuration="34.086880932s" totalImagesPullingTime="27.429392899s" totalInitContainerRuntime="3.722625131s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:09 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:43.084910719 +0000 UTC m=+547.702880221" watchObservedRunningTime="2026-09-11 16:26:43.086880932 +0000 UTC m=+547.704850435"
Sep 11 16:26:44.068247 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:44.068202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:26:44.069256 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:44.069228    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:26:44.072442 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:44.072422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:26:45.072577 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:45.072520    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:26:45.072994 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:45.072904    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:26:47.061305 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:47.061275    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:26:47.061785 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:47.061760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:26:55.072966 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:55.072914    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:26:55.073406 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:55.073335    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:26:57.062749 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:26:57.062707    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:27:05.073266 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:05.073217    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:27:05.073732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:05.073708    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:07.061894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:07.061857    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:27:15.073167 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:15.073112    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:27:15.073615 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:15.073552    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:17.062024 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:17.061989    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:27:25.072481 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:25.072426    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:27:25.072966 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:25.072881    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:27.062005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:27.061970    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:27:35.072814 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:35.072761    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:27:35.073241 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:35.073195    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:37.061701 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:37.061664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:27:45.072981 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:45.072928    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:27:45.073425 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:45.073406    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:46.891274 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:46.891233    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:27:46.892797 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:46.891506    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:47.061701 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:47.061655    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 11 16:27:56.892803 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:56.892718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:27:56.893378 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:56.893243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:27:57.063249 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:27:57.063217    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:28:04.921264 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:04.921232    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"]
Sep 11 16:28:04.921663 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:04.921517    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" containerID="cri-o://e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2" gracePeriod=30
Sep 11 16:28:04.921663 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:04.921561    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" containerID="cri-o://553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09" gracePeriod=30
Sep 11 16:28:04.921663 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:04.921571    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" containerID="cri-o://ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1" gracePeriod=30
Sep 11 16:28:05.420147 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:05.420113    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerID="553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09" exitCode=2
Sep 11 16:28:05.420291 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:05.420195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerDied","Data":"553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09"}
Sep 11 16:28:06.892257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:06.892215    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:06.893531 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:06.893501    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:28:09.068736 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:09.068681    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 11 16:28:10.443448 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:10.443414    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerID="e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2" exitCode=0
Sep 11 16:28:10.443809 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:10.443488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerDied","Data":"e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2"}
Sep 11 16:28:14.068470 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:14.068428    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 11 16:28:16.892557 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:16.892508    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:16.893754 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:16.893729    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:28:19.069033 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:19.068995    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 11 16:28:19.069486 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:19.069147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:28:24.069272 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:24.069228    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 11 16:28:26.891925 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:26.891882    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:28:26.892356 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:26.892034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:28:26.894111 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:26.894087    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 11 16:28:26.894203 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:26.894190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:28:29.069476 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:29.069430    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 11 16:28:34.068836 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:34.068791    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 11 16:28:35.112031 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.112000    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:28:35.144473 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144438    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-cabundle-cert\") pod \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") "
Sep 11 16:28:35.144655 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnrnk\" (UniqueName: \"kubernetes.io/projected/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-api-access-tnrnk\") pod \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") "
Sep 11 16:28:35.144655 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144562    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb518af5-500c-43d3-a3c8-5e81dcaf0977-proxy-tls\") pod \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") "
Sep 11 16:28:35.144655 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-rbac-proxy-sar-config\") pod \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") "
Sep 11 16:28:35.144655 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144650    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kserve-provision-location\") pod \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\" (UID: \"eb518af5-500c-43d3-a3c8-5e81dcaf0977\") "
Sep 11 16:28:35.144923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb518af5-500c-43d3-a3c8-5e81dcaf0977" (UID: "eb518af5-500c-43d3-a3c8-5e81dcaf0977"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:35.144923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.144896    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:28:35.145097 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.145075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb518af5-500c-43d3-a3c8-5e81dcaf0977" (UID: "eb518af5-500c-43d3-a3c8-5e81dcaf0977"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:28:35.145337 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.145303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb518af5-500c-43d3-a3c8-5e81dcaf0977" (UID: "eb518af5-500c-43d3-a3c8-5e81dcaf0977"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:35.147732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.147699    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb518af5-500c-43d3-a3c8-5e81dcaf0977-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb518af5-500c-43d3-a3c8-5e81dcaf0977" (UID: "eb518af5-500c-43d3-a3c8-5e81dcaf0977"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:28:35.147732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.147708    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-api-access-tnrnk" (OuterVolumeSpecName: "kube-api-access-tnrnk") pod "eb518af5-500c-43d3-a3c8-5e81dcaf0977" (UID: "eb518af5-500c-43d3-a3c8-5e81dcaf0977"). InnerVolumeSpecName "kube-api-access-tnrnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:35.245891 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.245800    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrnk\" (UniqueName: \"kubernetes.io/projected/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-api-access-tnrnk\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:28:35.245891 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.245832    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb518af5-500c-43d3-a3c8-5e81dcaf0977-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:28:35.245891 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.245841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:28:35.245891 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.245850    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb518af5-500c-43d3-a3c8-5e81dcaf0977-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:28:35.550033 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.549948    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerID="ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1" exitCode=0
Sep 11 16:28:35.550033 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.550004    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"
Sep 11 16:28:35.550033 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.550012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerDied","Data":"ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1"}
Sep 11 16:28:35.550255 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.550102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l" event={"ID":"eb518af5-500c-43d3-a3c8-5e81dcaf0977","Type":"ContainerDied","Data":"d81c7d381375225acc100a91c2d9e4f840b44db0345714dfe27b97cc94af1aed"}
Sep 11 16:28:35.550255 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.550137    2578 scope.go:117] "RemoveContainer" containerID="ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1"
Sep 11 16:28:35.558978 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.558950    2578 scope.go:117] "RemoveContainer" containerID="553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09"
Sep 11 16:28:35.567364 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.567344    2578 scope.go:117] "RemoveContainer" containerID="e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2"
Sep 11 16:28:35.571894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.571869    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"]
Sep 11 16:28:35.577165 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.576652    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a0b0b-predictor-79d7f65d76-ph47l"]
Sep 11 16:28:35.582005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.581982    2578 scope.go:117] "RemoveContainer" containerID="f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5"
Sep 11 16:28:35.590177 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.590152    2578 scope.go:117] "RemoveContainer" containerID="ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1"
Sep 11 16:28:35.590547 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:28:35.590527    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1\": container with ID starting with ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1 not found: ID does not exist" containerID="ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1"
Sep 11 16:28:35.590594 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.590557    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1"} err="failed to get container status \"ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1\": rpc error: code = NotFound desc = could not find container \"ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1\": container with ID starting with ceabbbe33023180373ed439b7bdb718d829607ee18aab7d2f32d6cff652837b1 not found: ID does not exist"
Sep 11 16:28:35.590594 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.590578    2578 scope.go:117] "RemoveContainer" containerID="553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09"
Sep 11 16:28:35.590882 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:28:35.590861    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09\": container with ID starting with 553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09 not found: ID does not exist" containerID="553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09"
Sep 11 16:28:35.590936 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.590888    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09"} err="failed to get container status \"553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09\": rpc error: code = NotFound desc = could not find container \"553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09\": container with ID starting with 553b68ec0fc06f27a26adc60fcbe939e970d2f07c23298ef92823da5f0201e09 not found: ID does not exist"
Sep 11 16:28:35.590936 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.590905    2578 scope.go:117] "RemoveContainer" containerID="e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2"
Sep 11 16:28:35.591208 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:28:35.591186    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2\": container with ID starting with e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2 not found: ID does not exist" containerID="e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2"
Sep 11 16:28:35.591266 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.591213    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2"} err="failed to get container status \"e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2\": rpc error: code = NotFound desc = could not find container \"e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2\": container with ID starting with e26c36f7ca2ead63f64e4f36dcf353863ebde69ccb2da432047fb249ccc7a8e2 not found: ID does not exist"
Sep 11 16:28:35.591266 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.591228    2578 scope.go:117] "RemoveContainer" containerID="f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5"
Sep 11 16:28:35.591641 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:28:35.591621    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5\": container with ID starting with f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5 not found: ID does not exist" containerID="f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5"
Sep 11 16:28:35.591724 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.591646    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5"} err="failed to get container status \"f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5\": rpc error: code = NotFound desc = could not find container \"f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5\": container with ID starting with f1d1098bbf180898643f11abc383cda64bf85562bab60010eca24692be3964c5 not found: ID does not exist"
Sep 11 16:28:35.896968 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:28:35.896935    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" path="/var/lib/kubelet/pods/eb518af5-500c-43d3-a3c8-5e81dcaf0977/volumes"
Sep 11 16:29:40.086255 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:40.086172    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"]
Sep 11 16:29:40.086758 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:40.086435    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container" containerID="cri-o://56ebe7eb2679486d380952eda9b3f0bb2b2c97e3a636edace2b52768f9060508" gracePeriod=30
Sep 11 16:29:40.086758 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:40.086482    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kube-rbac-proxy" containerID="cri-o://43a0f234d8a747b3e51752fc9d90d5d77a09dcfb617287104f9f4220479f8b69" gracePeriod=30
Sep 11 16:29:40.824687 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:40.824651    2578 generic.go:356] "Generic (PLEG): container finished" podID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerID="43a0f234d8a747b3e51752fc9d90d5d77a09dcfb617287104f9f4220479f8b69" exitCode=2
Sep 11 16:29:40.824857 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:40.824708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerDied","Data":"43a0f234d8a747b3e51752fc9d90d5d77a09dcfb617287104f9f4220479f8b69"}
Sep 11 16:29:42.056748 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:42.056704    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 11 16:29:44.844340 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.844303    2578 generic.go:356] "Generic (PLEG): container finished" podID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerID="56ebe7eb2679486d380952eda9b3f0bb2b2c97e3a636edace2b52768f9060508" exitCode=0
Sep 11 16:29:44.844702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.844363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerDied","Data":"56ebe7eb2679486d380952eda9b3f0bb2b2c97e3a636edace2b52768f9060508"}
Sep 11 16:29:44.940257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.940232    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:29:44.978661 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.978625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls\") pod \"81b8cd84-47ec-4163-8508-bd889ffe196c\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") "
Sep 11 16:29:44.978862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.978687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-cabundle-cert\") pod \"81b8cd84-47ec-4163-8508-bd889ffe196c\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") "
Sep 11 16:29:44.978862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.978747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kzkw\" (UniqueName: \"kubernetes.io/projected/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-api-access-2kzkw\") pod \"81b8cd84-47ec-4163-8508-bd889ffe196c\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") "
Sep 11 16:29:44.978862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.978780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-rbac-proxy-sar-config\") pod \"81b8cd84-47ec-4163-8508-bd889ffe196c\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") "
Sep 11 16:29:44.978862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.978820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b8cd84-47ec-4163-8508-bd889ffe196c-kserve-provision-location\") pod \"81b8cd84-47ec-4163-8508-bd889ffe196c\" (UID: \"81b8cd84-47ec-4163-8508-bd889ffe196c\") "
Sep 11 16:29:44.979158 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.979129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81b8cd84-47ec-4163-8508-bd889ffe196c" (UID: "81b8cd84-47ec-4163-8508-bd889ffe196c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:44.979217 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.979187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b8cd84-47ec-4163-8508-bd889ffe196c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81b8cd84-47ec-4163-8508-bd889ffe196c" (UID: "81b8cd84-47ec-4163-8508-bd889ffe196c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:29:44.979217 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.979191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81b8cd84-47ec-4163-8508-bd889ffe196c" (UID: "81b8cd84-47ec-4163-8508-bd889ffe196c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:44.980910 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.980880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81b8cd84-47ec-4163-8508-bd889ffe196c" (UID: "81b8cd84-47ec-4163-8508-bd889ffe196c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:44.981028 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:44.981010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-api-access-2kzkw" (OuterVolumeSpecName: "kube-api-access-2kzkw") pod "81b8cd84-47ec-4163-8508-bd889ffe196c" (UID: "81b8cd84-47ec-4163-8508-bd889ffe196c"). InnerVolumeSpecName "kube-api-access-2kzkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:29:45.080108 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.080004    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:29:45.080108 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.080037    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzkw\" (UniqueName: \"kubernetes.io/projected/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-api-access-2kzkw\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:29:45.080108 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.080080    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b8cd84-47ec-4163-8508-bd889ffe196c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:29:45.080108 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.080089    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b8cd84-47ec-4163-8508-bd889ffe196c-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:29:45.080108 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.080100    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b8cd84-47ec-4163-8508-bd889ffe196c-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:29:45.106967 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.106933    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"]
Sep 11 16:29:45.107278 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107266    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107281    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107291    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="storage-initializer"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107296    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="storage-initializer"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107306    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="storage-initializer"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107311    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="storage-initializer"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container"
Sep 11 16:29:45.107324 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107322    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107328    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kube-rbac-proxy"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107334    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kube-rbac-proxy"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107345    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107350    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107363    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107408    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kube-rbac-proxy"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107416    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="agent"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107424    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kserve-container"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107449    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" containerName="kserve-container"
Sep 11 16:29:45.107536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.107454    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb518af5-500c-43d3-a3c8-5e81dcaf0977" containerName="kube-rbac-proxy"
Sep 11 16:29:45.110654 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.110632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.113474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.113445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-77a5a-kube-rbac-proxy-sar-config\""
Sep 11 16:29:45.113615 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.113449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-77a5a-serving-cert\""
Sep 11 16:29:45.119097 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.119069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"]
Sep 11 16:29:45.180823 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.180781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-proxy-tls\") pod \"model-chainer-raw-77a5a-54c4649859-gq5v2\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.181127 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.180870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-openshift-service-ca-bundle\") pod \"model-chainer-raw-77a5a-54c4649859-gq5v2\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.282249 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.282217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-proxy-tls\") pod \"model-chainer-raw-77a5a-54c4649859-gq5v2\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.282461 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.282291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-openshift-service-ca-bundle\") pod \"model-chainer-raw-77a5a-54c4649859-gq5v2\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.282951 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.282930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-openshift-service-ca-bundle\") pod \"model-chainer-raw-77a5a-54c4649859-gq5v2\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.284775 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.284754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-proxy-tls\") pod \"model-chainer-raw-77a5a-54c4649859-gq5v2\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") " pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.298398 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.298356    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"]
Sep 11 16:29:45.304031 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.303992    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.306609 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.306586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"]
Sep 11 16:29:45.308032 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.308009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 11 16:29:45.308345 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.308067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 11 16:29:45.310307 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.310277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"]
Sep 11 16:29:45.310414 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.310400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.312676 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.312656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 11 16:29:45.324233 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.324208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"]
Sep 11 16:29:45.383635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.383635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.383848 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.383848 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkc2\" (UniqueName: \"kubernetes.io/projected/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-api-access-brkc2\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.383848 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2c17cea-f608-4186-b366-7f603eaed270-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.383848 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c17cea-f608-4186-b366-7f603eaed270-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.383973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a66f210-c969-4756-8746-b58e2e2f2de9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.383973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.383973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvpt\" (UniqueName: \"kubernetes.io/projected/e2c17cea-f608-4186-b366-7f603eaed270-kube-api-access-mtvpt\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.383973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.383926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.423170 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.423132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:45.484742 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a66f210-c969-4756-8746-b58e2e2f2de9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.484921 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.484921 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvpt\" (UniqueName: \"kubernetes.io/projected/e2c17cea-f608-4186-b366-7f603eaed270-kube-api-access-mtvpt\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.484921 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.484921 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.484921 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.484921 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.485276 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484961    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkc2\" (UniqueName: \"kubernetes.io/projected/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-api-access-brkc2\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.485276 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.484990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2c17cea-f608-4186-b366-7f603eaed270-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.485276 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.485147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c17cea-f608-4186-b366-7f603eaed270-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.485962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.485930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.485962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.485939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.486141 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:29:45.486077    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 11 16:29:45.486206 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:29:45.486149    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls podName:2a66f210-c969-4756-8746-b58e2e2f2de9 nodeName:}" failed. No retries permitted until 2026-09-11 16:29:45.986127147 +0000 UTC m=+730.604096642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls") pod "isvc-canary-stop-v2-predictor-596664c988-djj8t" (UID: "2a66f210-c969-4756-8746-b58e2e2f2de9") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 11 16:29:45.486993 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.486947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a66f210-c969-4756-8746-b58e2e2f2de9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.487365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.487327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c17cea-f608-4186-b366-7f603eaed270-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.487673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.487646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.487881 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.487849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.490340 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.490311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2c17cea-f608-4186-b366-7f603eaed270-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.498005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.497943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvpt\" (UniqueName: \"kubernetes.io/projected/e2c17cea-f608-4186-b366-7f603eaed270-kube-api-access-mtvpt\") pod \"isvc-canary-stop-predictor-5cdcf6545-4zg8t\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.498468 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.498443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkc2\" (UniqueName: \"kubernetes.io/projected/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-api-access-brkc2\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.626403 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.626364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:45.699383 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.699352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"]
Sep 11 16:29:45.701634 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:29:45.701605    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff0b447_271d_4a6a_9f02_ccdeee4000bc.slice/crio-749ff053dba892fec085650d0916bfcaa0d5cc529aa579a4f6749f27a076233a WatchSource:0}: Error finding container 749ff053dba892fec085650d0916bfcaa0d5cc529aa579a4f6749f27a076233a: Status 404 returned error can't find the container with id 749ff053dba892fec085650d0916bfcaa0d5cc529aa579a4f6749f27a076233a
Sep 11 16:29:45.703763 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.703747    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:29:45.851155 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.851121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj" event={"ID":"81b8cd84-47ec-4163-8508-bd889ffe196c","Type":"ContainerDied","Data":"03f0a5e7ee5323191784a01b0add69378b5d5cab1bad079a7387ce42ba881450"}
Sep 11 16:29:45.851629 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.851133    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"
Sep 11 16:29:45.851629 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.851197    2578 scope.go:117] "RemoveContainer" containerID="43a0f234d8a747b3e51752fc9d90d5d77a09dcfb617287104f9f4220479f8b69"
Sep 11 16:29:45.852660 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.852464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" event={"ID":"4ff0b447-271d-4a6a-9f02-ccdeee4000bc","Type":"ContainerStarted","Data":"749ff053dba892fec085650d0916bfcaa0d5cc529aa579a4f6749f27a076233a"}
Sep 11 16:29:45.860007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.859985    2578 scope.go:117] "RemoveContainer" containerID="56ebe7eb2679486d380952eda9b3f0bb2b2c97e3a636edace2b52768f9060508"
Sep 11 16:29:45.868249 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.868197    2578 scope.go:117] "RemoveContainer" containerID="36d3dbe69d2821f3f12cba2dfb9e043cf38b3884a3bae34c5e16c2cb27e2dd13"
Sep 11 16:29:45.875807 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.875777    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"]
Sep 11 16:29:45.880835 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.880809    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-snrpj"]
Sep 11 16:29:45.896544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.896462    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b8cd84-47ec-4163-8508-bd889ffe196c" path="/var/lib/kubelet/pods/81b8cd84-47ec-4163-8508-bd889ffe196c/volumes"
Sep 11 16:29:45.900619 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.900596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"]
Sep 11 16:29:45.903302 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:29:45.903277    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c17cea_f608_4186_b366_7f603eaed270.slice/crio-369fb1a9cac2598962c3833fb30d12a23d790a8af30beb58ba053a86d7e0d980 WatchSource:0}: Error finding container 369fb1a9cac2598962c3833fb30d12a23d790a8af30beb58ba053a86d7e0d980: Status 404 returned error can't find the container with id 369fb1a9cac2598962c3833fb30d12a23d790a8af30beb58ba053a86d7e0d980
Sep 11 16:29:45.990028 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.989994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:45.992563 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:45.992537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-djj8t\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:46.218602 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:46.218506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:46.494617 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:46.494592    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"]
Sep 11 16:29:46.497265 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:29:46.497239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a66f210_c969_4756_8746_b58e2e2f2de9.slice/crio-c98550ac612c5c56235eeb6eedff328cec205c38c9baa6b2b818a90777ba7b8a WatchSource:0}: Error finding container c98550ac612c5c56235eeb6eedff328cec205c38c9baa6b2b818a90777ba7b8a: Status 404 returned error can't find the container with id c98550ac612c5c56235eeb6eedff328cec205c38c9baa6b2b818a90777ba7b8a
Sep 11 16:29:46.860982 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:46.860758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerStarted","Data":"df02317929f0d08104e74da409deeed79ebe0c2753b0760c7519af45ccb0d2ef"}
Sep 11 16:29:46.860982 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:46.860842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerStarted","Data":"c98550ac612c5c56235eeb6eedff328cec205c38c9baa6b2b818a90777ba7b8a"}
Sep 11 16:29:46.862564 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:46.862536    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerStarted","Data":"b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f"}
Sep 11 16:29:46.862679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:46.862596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerStarted","Data":"369fb1a9cac2598962c3833fb30d12a23d790a8af30beb58ba053a86d7e0d980"}
Sep 11 16:29:48.873948 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:48.873914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" event={"ID":"4ff0b447-271d-4a6a-9f02-ccdeee4000bc","Type":"ContainerStarted","Data":"42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180"}
Sep 11 16:29:48.874362 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:48.874071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:48.892836 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:48.892788    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podStartSLOduration=1.721323138 podStartE2EDuration="3.892772949s" totalImagesPullingTime="2.171449811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:48.891494357 +0000 UTC m=+733.509463839" watchObservedRunningTime="2026-09-11 16:29:48.892772949 +0000 UTC m=+733.510742449"
Sep 11 16:29:50.884900 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:50.884867    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerID="df02317929f0d08104e74da409deeed79ebe0c2753b0760c7519af45ccb0d2ef" exitCode=0
Sep 11 16:29:50.885384 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:50.884943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerDied","Data":"df02317929f0d08104e74da409deeed79ebe0c2753b0760c7519af45ccb0d2ef"}
Sep 11 16:29:50.886584 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:50.886563    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2c17cea-f608-4186-b366-7f603eaed270" containerID="b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f" exitCode=0
Sep 11 16:29:50.886680 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:50.886624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerDied","Data":"b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f"}
Sep 11 16:29:51.895878 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.895844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerStarted","Data":"d1e427fa548f541163dded30b127dd9734276331af831bcbb280ad6fc23c45ac"}
Sep 11 16:29:51.896301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.895907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerStarted","Data":"bb119cb0907f83f5e9a805bbabd6dda0c68b32596e3837c634a839ec9e55c540"}
Sep 11 16:29:51.896301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.896093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerStarted","Data":"880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c"}
Sep 11 16:29:51.896301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.896150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerStarted","Data":"bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262"}
Sep 11 16:29:51.896301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.896192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:51.896487 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.896409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:51.896487 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.896464    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:51.897786 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.897760    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:29:51.917850 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.917789    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podStartSLOduration=2.789926745 podStartE2EDuration="6.917766513s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.127839768s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:51.917658332 +0000 UTC m=+736.535627874" watchObservedRunningTime="2026-09-11 16:29:51.917766513 +0000 UTC m=+736.535736015"
Sep 11 16:29:51.935176 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:51.935120    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podStartSLOduration=3.017944011 podStartE2EDuration="6.93510357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.917159559s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:51.934031367 +0000 UTC m=+736.552000868" watchObservedRunningTime="2026-09-11 16:29:51.93510357 +0000 UTC m=+736.553073070"
Sep 11 16:29:52.901037 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:52.900993    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:29:52.901487 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:52.901178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:52.902470 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:52.902448    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:29:53.905577 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:53.905536    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:29:54.886018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:54.885986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:29:55.138113 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:55.138011    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"]
Sep 11 16:29:55.138704 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:55.138261    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" containerID="cri-o://42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180" gracePeriod=30
Sep 11 16:29:57.905505 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:57.905478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:29:57.906019 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:57.905993    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:29:58.910725 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:58.910693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:29:58.911317 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:58.911287    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:29:59.885041 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:29:59.884994    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:04.884459 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:04.884408    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:07.906818 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:07.906772    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:30:08.911425 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:08.911384    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:30:09.884006 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:09.883967    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:09.884193 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:09.884112    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:30:14.883062 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:14.883000    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:17.906121 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:17.906086    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:30:18.911851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:18.911809    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:30:19.884720 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:19.884672    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:24.883161 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:24.883117    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:30:25.291588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.291561    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:30:25.420586 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.420503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-proxy-tls\") pod \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") "
Sep 11 16:30:25.420586 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.420556    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-openshift-service-ca-bundle\") pod \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\" (UID: \"4ff0b447-271d-4a6a-9f02-ccdeee4000bc\") "
Sep 11 16:30:25.420964 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.420938    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4ff0b447-271d-4a6a-9f02-ccdeee4000bc" (UID: "4ff0b447-271d-4a6a-9f02-ccdeee4000bc"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:30:25.422817 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.422794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ff0b447-271d-4a6a-9f02-ccdeee4000bc" (UID: "4ff0b447-271d-4a6a-9f02-ccdeee4000bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:30:25.522042 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.522001    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:30:25.522042 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:25.522036    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ff0b447-271d-4a6a-9f02-ccdeee4000bc-openshift-service-ca-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:30:26.051837 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.051791    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerID="42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180" exitCode=0
Sep 11 16:30:26.052335 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.051854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"
Sep 11 16:30:26.052335 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.051872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" event={"ID":"4ff0b447-271d-4a6a-9f02-ccdeee4000bc","Type":"ContainerDied","Data":"42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180"}
Sep 11 16:30:26.052335 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.051926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2" event={"ID":"4ff0b447-271d-4a6a-9f02-ccdeee4000bc","Type":"ContainerDied","Data":"749ff053dba892fec085650d0916bfcaa0d5cc529aa579a4f6749f27a076233a"}
Sep 11 16:30:26.052335 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.051948    2578 scope.go:117] "RemoveContainer" containerID="42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180"
Sep 11 16:30:26.060413 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.060397    2578 scope.go:117] "RemoveContainer" containerID="42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180"
Sep 11 16:30:26.060715 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:30:26.060696    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180\": container with ID starting with 42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180 not found: ID does not exist" containerID="42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180"
Sep 11 16:30:26.060767 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.060723    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180"} err="failed to get container status \"42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180\": rpc error: code = NotFound desc = could not find container \"42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180\": container with ID starting with 42040be36f1a5a288506caed4a2e2ce08909af77be8b2e757072b6ffa6817180 not found: ID does not exist"
Sep 11 16:30:26.068424 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.068361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"]
Sep 11 16:30:26.069927 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:26.069906    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-77a5a-54c4649859-gq5v2"]
Sep 11 16:30:27.895223 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:27.895196    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" path="/var/lib/kubelet/pods/4ff0b447-271d-4a6a-9f02-ccdeee4000bc/volumes"
Sep 11 16:30:27.906111 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:27.906079    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:30:28.911590 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:28.911554    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:30:37.906112 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:37.906074    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:30:38.912097 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:38.912028    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:30:47.905911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:47.905868    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:30:48.911728 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:48.911686    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:30:57.906040 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:57.905953    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:30:58.912037 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:30:58.911996    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:31:03.890988 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:03.890945    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 11 16:31:08.912768 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:08.912737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:31:09.037320 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:09.037285    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"]
Sep 11 16:31:09.037593 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:09.037562    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container" containerID="cri-o://bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262" gracePeriod=30
Sep 11 16:31:09.037751 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:09.037627    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kube-rbac-proxy" containerID="cri-o://880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c" gracePeriod=30
Sep 11 16:31:09.038495 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:09.038429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:31:09.237764 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:09.237676    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2c17cea-f608-4186-b366-7f603eaed270" containerID="880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c" exitCode=2
Sep 11 16:31:09.237899 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:09.237754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerDied","Data":"880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c"}
Sep 11 16:31:12.902222 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:12.902178    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 11 16:31:13.873018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:13.872993    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:31:14.021320 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-cabundle-cert\") pod \"e2c17cea-f608-4186-b366-7f603eaed270\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") "
Sep 11 16:31:14.021320 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021271    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2c17cea-f608-4186-b366-7f603eaed270-proxy-tls\") pod \"e2c17cea-f608-4186-b366-7f603eaed270\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") "
Sep 11 16:31:14.021320 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtvpt\" (UniqueName: \"kubernetes.io/projected/e2c17cea-f608-4186-b366-7f603eaed270-kube-api-access-mtvpt\") pod \"e2c17cea-f608-4186-b366-7f603eaed270\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") "
Sep 11 16:31:14.021843 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c17cea-f608-4186-b366-7f603eaed270-kserve-provision-location\") pod \"e2c17cea-f608-4186-b366-7f603eaed270\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") "
Sep 11 16:31:14.021843 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-kube-rbac-proxy-sar-config\") pod \"e2c17cea-f608-4186-b366-7f603eaed270\" (UID: \"e2c17cea-f608-4186-b366-7f603eaed270\") "
Sep 11 16:31:14.021843 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c17cea-f608-4186-b366-7f603eaed270-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2c17cea-f608-4186-b366-7f603eaed270" (UID: "e2c17cea-f608-4186-b366-7f603eaed270"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:31:14.021843 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2c17cea-f608-4186-b366-7f603eaed270" (UID: "e2c17cea-f608-4186-b366-7f603eaed270"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:14.021843 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.021788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2c17cea-f608-4186-b366-7f603eaed270" (UID: "e2c17cea-f608-4186-b366-7f603eaed270"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:14.023532 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.023509    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c17cea-f608-4186-b366-7f603eaed270-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2c17cea-f608-4186-b366-7f603eaed270" (UID: "e2c17cea-f608-4186-b366-7f603eaed270"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:31:14.023635 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.023608    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c17cea-f608-4186-b366-7f603eaed270-kube-api-access-mtvpt" (OuterVolumeSpecName: "kube-api-access-mtvpt") pod "e2c17cea-f608-4186-b366-7f603eaed270" (UID: "e2c17cea-f608-4186-b366-7f603eaed270"). InnerVolumeSpecName "kube-api-access-mtvpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:14.122318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.122280    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2c17cea-f608-4186-b366-7f603eaed270-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:14.122318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.122313    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:14.122318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.122323    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2c17cea-f608-4186-b366-7f603eaed270-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:14.122554 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.122333    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2c17cea-f608-4186-b366-7f603eaed270-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:14.122554 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.122341    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtvpt\" (UniqueName: \"kubernetes.io/projected/e2c17cea-f608-4186-b366-7f603eaed270-kube-api-access-mtvpt\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:14.259144 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.259105    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2c17cea-f608-4186-b366-7f603eaed270" containerID="bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262" exitCode=0
Sep 11 16:31:14.259314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.259169    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"
Sep 11 16:31:14.259314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.259187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerDied","Data":"bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262"}
Sep 11 16:31:14.259314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.259254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t" event={"ID":"e2c17cea-f608-4186-b366-7f603eaed270","Type":"ContainerDied","Data":"369fb1a9cac2598962c3833fb30d12a23d790a8af30beb58ba053a86d7e0d980"}
Sep 11 16:31:14.259314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.259282    2578 scope.go:117] "RemoveContainer" containerID="880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c"
Sep 11 16:31:14.267600 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.267580    2578 scope.go:117] "RemoveContainer" containerID="bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262"
Sep 11 16:31:14.275009 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.274991    2578 scope.go:117] "RemoveContainer" containerID="b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f"
Sep 11 16:31:14.282710 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.282688    2578 scope.go:117] "RemoveContainer" containerID="880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c"
Sep 11 16:31:14.283033 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:31:14.283013    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c\": container with ID starting with 880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c not found: ID does not exist" containerID="880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c"
Sep 11 16:31:14.283147 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.283125    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c"} err="failed to get container status \"880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c\": rpc error: code = NotFound desc = could not find container \"880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c\": container with ID starting with 880b263a9690b2cbddc81bab165c05bb749c0b1a788a4af1ff9b25e66a060d0c not found: ID does not exist"
Sep 11 16:31:14.283203 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.283155    2578 scope.go:117] "RemoveContainer" containerID="bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262"
Sep 11 16:31:14.283203 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.283136    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"]
Sep 11 16:31:14.283467 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:31:14.283438    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262\": container with ID starting with bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262 not found: ID does not exist" containerID="bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262"
Sep 11 16:31:14.283548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.283465    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262"} err="failed to get container status \"bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262\": rpc error: code = NotFound desc = could not find container \"bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262\": container with ID starting with bc45db0744632d36faac78fa30e1a126d8d75149ceae32094a21e59bef953262 not found: ID does not exist"
Sep 11 16:31:14.283548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.283498    2578 scope.go:117] "RemoveContainer" containerID="b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f"
Sep 11 16:31:14.283793 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:31:14.283777    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f\": container with ID starting with b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f not found: ID does not exist" containerID="b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f"
Sep 11 16:31:14.283828 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.283799    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f"} err="failed to get container status \"b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f\": rpc error: code = NotFound desc = could not find container \"b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f\": container with ID starting with b5c1735d02b31ddd5934370a4cba785e8affd4f39b11bd7ad6518ed9fe6ea31f not found: ID does not exist"
Sep 11 16:31:14.286420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:14.286399    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-4zg8t"]
Sep 11 16:31:15.258028 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:15.257990    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"]
Sep 11 16:31:15.258431 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:15.258274    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" containerID="cri-o://bb119cb0907f83f5e9a805bbabd6dda0c68b32596e3837c634a839ec9e55c540" gracePeriod=30
Sep 11 16:31:15.258431 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:15.258316    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kube-rbac-proxy" containerID="cri-o://d1e427fa548f541163dded30b127dd9734276331af831bcbb280ad6fc23c45ac" gracePeriod=30
Sep 11 16:31:15.895144 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:15.895116    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c17cea-f608-4186-b366-7f603eaed270" path="/var/lib/kubelet/pods/e2c17cea-f608-4186-b366-7f603eaed270/volumes"
Sep 11 16:31:16.272884 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:16.272804    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerID="d1e427fa548f541163dded30b127dd9734276331af831bcbb280ad6fc23c45ac" exitCode=2
Sep 11 16:31:16.273237 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:16.272878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerDied","Data":"d1e427fa548f541163dded30b127dd9734276331af831bcbb280ad6fc23c45ac"}
Sep 11 16:31:18.906343 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:18.906302    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 11 16:31:18.911668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:18.911629    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 16:31:20.298928 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.298894    2578 generic.go:356] "Generic (PLEG): container finished" podID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerID="bb119cb0907f83f5e9a805bbabd6dda0c68b32596e3837c634a839ec9e55c540" exitCode=0
Sep 11 16:31:20.299339 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.298968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerDied","Data":"bb119cb0907f83f5e9a805bbabd6dda0c68b32596e3837c634a839ec9e55c540"}
Sep 11 16:31:20.320005 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.319981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:31:20.473828 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.473790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a66f210-c969-4756-8746-b58e2e2f2de9-kserve-provision-location\") pod \"2a66f210-c969-4756-8746-b58e2e2f2de9\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") "
Sep 11 16:31:20.474022 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.473841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls\") pod \"2a66f210-c969-4756-8746-b58e2e2f2de9\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") "
Sep 11 16:31:20.474022 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.473883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-rbac-proxy-sar-config\") pod \"2a66f210-c969-4756-8746-b58e2e2f2de9\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") "
Sep 11 16:31:20.474022 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.473898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-cabundle-cert\") pod \"2a66f210-c969-4756-8746-b58e2e2f2de9\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") "
Sep 11 16:31:20.474022 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.473934    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brkc2\" (UniqueName: \"kubernetes.io/projected/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-api-access-brkc2\") pod \"2a66f210-c969-4756-8746-b58e2e2f2de9\" (UID: \"2a66f210-c969-4756-8746-b58e2e2f2de9\") "
Sep 11 16:31:20.474284 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.474214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a66f210-c969-4756-8746-b58e2e2f2de9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a66f210-c969-4756-8746-b58e2e2f2de9" (UID: "2a66f210-c969-4756-8746-b58e2e2f2de9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:31:20.474339 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.474285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a66f210-c969-4756-8746-b58e2e2f2de9" (UID: "2a66f210-c969-4756-8746-b58e2e2f2de9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:20.474339 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.474290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a66f210-c969-4756-8746-b58e2e2f2de9" (UID: "2a66f210-c969-4756-8746-b58e2e2f2de9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:31:20.476285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.476262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a66f210-c969-4756-8746-b58e2e2f2de9" (UID: "2a66f210-c969-4756-8746-b58e2e2f2de9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:31:20.476368 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.476262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-api-access-brkc2" (OuterVolumeSpecName: "kube-api-access-brkc2") pod "2a66f210-c969-4756-8746-b58e2e2f2de9" (UID: "2a66f210-c969-4756-8746-b58e2e2f2de9"). InnerVolumeSpecName "kube-api-access-brkc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:31:20.574606 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.574561    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a66f210-c969-4756-8746-b58e2e2f2de9-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.574606 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.574596    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a66f210-c969-4756-8746-b58e2e2f2de9-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.574606 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.574606    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.574606 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.574616    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a66f210-c969-4756-8746-b58e2e2f2de9-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:20.574863 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:20.574625    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkc2\" (UniqueName: \"kubernetes.io/projected/2a66f210-c969-4756-8746-b58e2e2f2de9-kube-api-access-brkc2\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:31:21.305263 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.305237    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"
Sep 11 16:31:21.305797 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.305228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t" event={"ID":"2a66f210-c969-4756-8746-b58e2e2f2de9","Type":"ContainerDied","Data":"c98550ac612c5c56235eeb6eedff328cec205c38c9baa6b2b818a90777ba7b8a"}
Sep 11 16:31:21.305797 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.305397    2578 scope.go:117] "RemoveContainer" containerID="d1e427fa548f541163dded30b127dd9734276331af831bcbb280ad6fc23c45ac"
Sep 11 16:31:21.313708 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.313688    2578 scope.go:117] "RemoveContainer" containerID="bb119cb0907f83f5e9a805bbabd6dda0c68b32596e3837c634a839ec9e55c540"
Sep 11 16:31:21.321885 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.321865    2578 scope.go:117] "RemoveContainer" containerID="df02317929f0d08104e74da409deeed79ebe0c2753b0760c7519af45ccb0d2ef"
Sep 11 16:31:21.326750 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.326727    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"]
Sep 11 16:31:21.330567 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.330542    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-djj8t"]
Sep 11 16:31:21.895023 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:21.894983    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" path="/var/lib/kubelet/pods/2a66f210-c969-4756-8746-b58e2e2f2de9/volumes"
Sep 11 16:31:25.531128 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531096    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"]
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531378    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531388    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531397    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kube-rbac-proxy"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kube-rbac-proxy"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531413    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="storage-initializer"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531418    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="storage-initializer"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531430    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kube-rbac-proxy"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531434    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kube-rbac-proxy"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531442    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531447    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="storage-initializer"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531458    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="storage-initializer"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531463    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container"
Sep 11 16:31:25.531502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531468    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container"
Sep 11 16:31:25.531914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531540    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kserve-container"
Sep 11 16:31:25.531914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531549    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff0b447-271d-4a6a-9f02-ccdeee4000bc" containerName="model-chainer-raw-77a5a"
Sep 11 16:31:25.531914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531554    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c17cea-f608-4186-b366-7f603eaed270" containerName="kube-rbac-proxy"
Sep 11 16:31:25.531914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531559    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kube-rbac-proxy"
Sep 11 16:31:25.531914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.531565    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a66f210-c969-4756-8746-b58e2e2f2de9" containerName="kserve-container"
Sep 11 16:31:25.535463 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.535443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.538207 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.538181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:31:25.538360 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.538252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d91b1e-kube-rbac-proxy-sar-config\""
Sep 11 16:31:25.538360 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.538292    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:31:25.538472 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.538408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:31:25.539274 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.539257    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:31:25.539398 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.539352    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d91b1e-predictor-serving-cert\""
Sep 11 16:31:25.539398 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.539332    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:31:25.545781 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.545752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"]
Sep 11 16:31:25.717167 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.717114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90cba922-3d57-4b1f-9492-9cfc3ed204d1-proxy-tls\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.717167 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.717172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-cabundle-cert\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.717353 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.717237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744wq\" (UniqueName: \"kubernetes.io/projected/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-api-access-744wq\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.717353 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.717276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kserve-provision-location\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.717353 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.717318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.817919 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.817827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kserve-provision-location\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.817919 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.817894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.817919 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.817919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90cba922-3d57-4b1f-9492-9cfc3ed204d1-proxy-tls\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.818188 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.817947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-cabundle-cert\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.818188 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.817969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744wq\" (UniqueName: \"kubernetes.io/projected/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-api-access-744wq\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.818291 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.818261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kserve-provision-location\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.818529 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.818509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-cabundle-cert\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.818590 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.818565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.820539 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.820516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90cba922-3d57-4b1f-9492-9cfc3ed204d1-proxy-tls\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.826599 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.826573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-744wq\" (UniqueName: \"kubernetes.io/projected/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-api-access-744wq\") pod \"isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") " pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:25.848373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:25.848336    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:26.120652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:26.120619    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"]
Sep 11 16:31:26.124321 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:31:26.124293    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cba922_3d57_4b1f_9492_9cfc3ed204d1.slice/crio-e00d899d0aed82f50699244ae11c66d8bb58f2ac9a776d25dab51f2e9d68ceba WatchSource:0}: Error finding container e00d899d0aed82f50699244ae11c66d8bb58f2ac9a776d25dab51f2e9d68ceba: Status 404 returned error can't find the container with id e00d899d0aed82f50699244ae11c66d8bb58f2ac9a776d25dab51f2e9d68ceba
Sep 11 16:31:26.330779 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:26.330741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerStarted","Data":"08b080648de182564828576696b89c8e4ff04fe1ec5ff94049fb5a3a9ebf4532"}
Sep 11 16:31:26.330952 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:26.330799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerStarted","Data":"e00d899d0aed82f50699244ae11c66d8bb58f2ac9a776d25dab51f2e9d68ceba"}
Sep 11 16:31:30.350017 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:30.349930    2578 generic.go:356] "Generic (PLEG): container finished" podID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerID="08b080648de182564828576696b89c8e4ff04fe1ec5ff94049fb5a3a9ebf4532" exitCode=0
Sep 11 16:31:30.350017 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:30.350004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerDied","Data":"08b080648de182564828576696b89c8e4ff04fe1ec5ff94049fb5a3a9ebf4532"}
Sep 11 16:31:31.357012 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:31.356979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerStarted","Data":"021908dbed5a383817a2d5eaf37a36e0d93169fd0c0c915d57610b06e8c91e8a"}
Sep 11 16:31:31.357410 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:31.357062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerStarted","Data":"0c14bd6636dc1881b3ed656dfa2647e85a365846af62271941edaf49f1770b19"}
Sep 11 16:31:31.357410 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:31.357259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:31.377452 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:31.377398    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podStartSLOduration=2.67922954 podStartE2EDuration="6.377378035s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698148495s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:31.376151827 +0000 UTC m=+835.994121330" watchObservedRunningTime="2026-09-11 16:31:31.377378035 +0000 UTC m=+835.995347565"
Sep 11 16:31:32.361933 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:32.361900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:32.363166 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:32.363138    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:31:33.365819 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:33.365782    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:31:35.405353 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.405323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"]
Sep 11 16:31:35.408099 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.408075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.410611 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.410589    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-785db-serving-cert\""
Sep 11 16:31:35.410741 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.410597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-785db-kube-rbac-proxy-sar-config\""
Sep 11 16:31:35.416227 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.416205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"]
Sep 11 16:31:35.491726 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.491688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec4be61-426c-4012-8017-4b3d0afc3250-proxy-tls\") pod \"model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.491888 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.491740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec4be61-426c-4012-8017-4b3d0afc3250-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.592225 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.592186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec4be61-426c-4012-8017-4b3d0afc3250-proxy-tls\") pod \"model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.592396 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.592234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec4be61-426c-4012-8017-4b3d0afc3250-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.592976 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.592932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec4be61-426c-4012-8017-4b3d0afc3250-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.594780 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.594758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec4be61-426c-4012-8017-4b3d0afc3250-proxy-tls\") pod \"model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.719159 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.719067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:35.988993 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:35.988966    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"]
Sep 11 16:31:35.991325 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:31:35.991296    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec4be61_426c_4012_8017_4b3d0afc3250.slice/crio-d5269d06d0949202d7b3b3ec8355179ba508818932b7fc6dc6cc643efbadb0de WatchSource:0}: Error finding container d5269d06d0949202d7b3b3ec8355179ba508818932b7fc6dc6cc643efbadb0de: Status 404 returned error can't find the container with id d5269d06d0949202d7b3b3ec8355179ba508818932b7fc6dc6cc643efbadb0de
Sep 11 16:31:36.381348 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:36.381309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" event={"ID":"1ec4be61-426c-4012-8017-4b3d0afc3250","Type":"ContainerStarted","Data":"75444cfdbc3d3548fcdb68489774b4b3c4e6371a806f0de8cd5c8bc46bfe1908"}
Sep 11 16:31:36.381581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:36.381369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" event={"ID":"1ec4be61-426c-4012-8017-4b3d0afc3250","Type":"ContainerStarted","Data":"d5269d06d0949202d7b3b3ec8355179ba508818932b7fc6dc6cc643efbadb0de"}
Sep 11 16:31:36.381581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:36.381422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:36.396947 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:36.396900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podStartSLOduration=1.396883386 podStartE2EDuration="1.396883386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:36.396337136 +0000 UTC m=+841.014306631" watchObservedRunningTime="2026-09-11 16:31:36.396883386 +0000 UTC m=+841.014852885"
Sep 11 16:31:38.370209 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:38.370178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:31:38.370698 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:38.370673    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:31:42.391623 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:42.391592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:45.466633 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.466596    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"]
Sep 11 16:31:45.467112 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.466817    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" containerID="cri-o://75444cfdbc3d3548fcdb68489774b4b3c4e6371a806f0de8cd5c8bc46bfe1908" gracePeriod=30
Sep 11 16:31:45.662748 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.662717    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"]
Sep 11 16:31:45.665178 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.665160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.667806 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.667783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-ecc5e-predictor-serving-cert\""
Sep 11 16:31:45.667928 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.667889    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-ecc5e-kube-rbac-proxy-sar-config\""
Sep 11 16:31:45.689329 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.689294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"]
Sep 11 16:31:45.758489 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.758403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.758489 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.758448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qnd\" (UniqueName: \"kubernetes.io/projected/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-api-access-k7qnd\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.758669 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.758531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.859534 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.859494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.859744 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.859549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qnd\" (UniqueName: \"kubernetes.io/projected/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-api-access-k7qnd\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.859744 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.859583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.859744 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:31:45.859648    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-serving-cert: secret "message-dumper-raw-ecc5e-predictor-serving-cert" not found
Sep 11 16:31:45.859744 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:31:45.859727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls podName:fa54cfd9-b196-400e-9239-3682f7aae5ab nodeName:}" failed. No retries permitted until 2026-09-11 16:31:46.359706403 +0000 UTC m=+850.977675896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls") pod "message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" (UID: "fa54cfd9-b196-400e-9239-3682f7aae5ab") : secret "message-dumper-raw-ecc5e-predictor-serving-cert" not found
Sep 11 16:31:45.860203 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.860187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:45.868593 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:45.868562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qnd\" (UniqueName: \"kubernetes.io/projected/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-api-access-k7qnd\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:46.363594 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:46.363555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:46.366086 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:46.366064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls\") pod \"message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") " pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:46.577665 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:46.577622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:46.851023 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:46.850989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"]
Sep 11 16:31:46.853930 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:31:46.853902    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa54cfd9_b196_400e_9239_3682f7aae5ab.slice/crio-7be43d1467645a369fdf7c3dbab213cff973aaf594cb4b0884c9ccdbb3cfc905 WatchSource:0}: Error finding container 7be43d1467645a369fdf7c3dbab213cff973aaf594cb4b0884c9ccdbb3cfc905: Status 404 returned error can't find the container with id 7be43d1467645a369fdf7c3dbab213cff973aaf594cb4b0884c9ccdbb3cfc905
Sep 11 16:31:47.389831 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:47.389796    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:47.427094 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:47.427038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" event={"ID":"fa54cfd9-b196-400e-9239-3682f7aae5ab","Type":"ContainerStarted","Data":"7be43d1467645a369fdf7c3dbab213cff973aaf594cb4b0884c9ccdbb3cfc905"}
Sep 11 16:31:48.370727 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.370686    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:31:48.432969 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.432934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" event={"ID":"fa54cfd9-b196-400e-9239-3682f7aae5ab","Type":"ContainerStarted","Data":"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"}
Sep 11 16:31:48.433186 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.433039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" event={"ID":"fa54cfd9-b196-400e-9239-3682f7aae5ab","Type":"ContainerStarted","Data":"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"}
Sep 11 16:31:48.433186 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.433095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:48.433186 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.433124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:48.435037 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.435015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:48.451797 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:48.451745    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" podStartSLOduration=2.224733467 podStartE2EDuration="3.451725167s" totalImagesPullingTime="1.2269917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:48.451020862 +0000 UTC m=+853.068990362" watchObservedRunningTime="2026-09-11 16:31:48.451725167 +0000 UTC m=+853.069694667"
Sep 11 16:31:52.390135 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:52.390090    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:55.445017 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:55.444986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:31:57.389847 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:57.389810    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:31:57.390252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:57.389921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:31:58.371698 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:31:58.371654    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:32:02.390730 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:02.390690    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:07.390065 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:07.389998    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:08.371301 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:08.371260    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:32:12.389700 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:12.389655    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:32:15.551168 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.551134    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerID="75444cfdbc3d3548fcdb68489774b4b3c4e6371a806f0de8cd5c8bc46bfe1908" exitCode=0
Sep 11 16:32:15.551533 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.551200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" event={"ID":"1ec4be61-426c-4012-8017-4b3d0afc3250","Type":"ContainerDied","Data":"75444cfdbc3d3548fcdb68489774b4b3c4e6371a806f0de8cd5c8bc46bfe1908"}
Sep 11 16:32:15.612688 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.612661    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:32:15.698170 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.698128    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec4be61-426c-4012-8017-4b3d0afc3250-openshift-service-ca-bundle\") pod \"1ec4be61-426c-4012-8017-4b3d0afc3250\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") "
Sep 11 16:32:15.698344 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.698189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec4be61-426c-4012-8017-4b3d0afc3250-proxy-tls\") pod \"1ec4be61-426c-4012-8017-4b3d0afc3250\" (UID: \"1ec4be61-426c-4012-8017-4b3d0afc3250\") "
Sep 11 16:32:15.698490 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.698462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec4be61-426c-4012-8017-4b3d0afc3250-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1ec4be61-426c-4012-8017-4b3d0afc3250" (UID: "1ec4be61-426c-4012-8017-4b3d0afc3250"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:32:15.700483 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.700454    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec4be61-426c-4012-8017-4b3d0afc3250-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ec4be61-426c-4012-8017-4b3d0afc3250" (UID: "1ec4be61-426c-4012-8017-4b3d0afc3250"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:32:15.799481 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.799398    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ec4be61-426c-4012-8017-4b3d0afc3250-openshift-service-ca-bundle\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:32:15.799481 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:15.799424    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ec4be61-426c-4012-8017-4b3d0afc3250-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:32:16.556847 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:16.556819    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"
Sep 11 16:32:16.557343 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:16.556846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz" event={"ID":"1ec4be61-426c-4012-8017-4b3d0afc3250","Type":"ContainerDied","Data":"d5269d06d0949202d7b3b3ec8355179ba508818932b7fc6dc6cc643efbadb0de"}
Sep 11 16:32:16.557343 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:16.556909    2578 scope.go:117] "RemoveContainer" containerID="75444cfdbc3d3548fcdb68489774b4b3c4e6371a806f0de8cd5c8bc46bfe1908"
Sep 11 16:32:16.575550 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:16.575500    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"]
Sep 11 16:32:16.577679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:16.577654    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-785db-7c8fcd7f4d-w9lqz"]
Sep 11 16:32:17.895153 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:17.895124    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" path="/var/lib/kubelet/pods/1ec4be61-426c-4012-8017-4b3d0afc3250/volumes"
Sep 11 16:32:18.371330 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:18.371291    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:32:28.371327 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:28.371239    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:32:38.370671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:38.370629    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:32:43.891533 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:43.891489    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 11 16:32:53.895971 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:32:53.895943    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:33:28.788569 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:28.788534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"]
Sep 11 16:33:28.789396 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:28.788800    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" containerID="cri-o://0c14bd6636dc1881b3ed656dfa2647e85a365846af62271941edaf49f1770b19" gracePeriod=30
Sep 11 16:33:28.789396 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:28.788876    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kube-rbac-proxy" containerID="cri-o://021908dbed5a383817a2d5eaf37a36e0d93169fd0c0c915d57610b06e8c91e8a" gracePeriod=30
Sep 11 16:33:29.876766 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:29.876733    2578 generic.go:356] "Generic (PLEG): container finished" podID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerID="021908dbed5a383817a2d5eaf37a36e0d93169fd0c0c915d57610b06e8c91e8a" exitCode=2
Sep 11 16:33:29.877167 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:29.876780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerDied","Data":"021908dbed5a383817a2d5eaf37a36e0d93169fd0c0c915d57610b06e8c91e8a"}
Sep 11 16:33:33.366815 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:33.366773    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 11 16:33:33.900749 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:33.900716    2578 generic.go:356] "Generic (PLEG): container finished" podID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerID="0c14bd6636dc1881b3ed656dfa2647e85a365846af62271941edaf49f1770b19" exitCode=0
Sep 11 16:33:33.900902 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:33.900805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerDied","Data":"0c14bd6636dc1881b3ed656dfa2647e85a365846af62271941edaf49f1770b19"}
Sep 11 16:33:33.957213 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:33.957184    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:33:34.024337 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024298    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-744wq\" (UniqueName: \"kubernetes.io/projected/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-api-access-744wq\") pod \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") "
Sep 11 16:33:34.024504 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-rbac-proxy-sar-config\") pod \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") "
Sep 11 16:33:34.024504 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kserve-provision-location\") pod \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") "
Sep 11 16:33:34.024504 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90cba922-3d57-4b1f-9492-9cfc3ed204d1-proxy-tls\") pod \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") "
Sep 11 16:33:34.024504 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-cabundle-cert\") pod \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\" (UID: \"90cba922-3d57-4b1f-9492-9cfc3ed204d1\") "
Sep 11 16:33:34.024849 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024714    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90cba922-3d57-4b1f-9492-9cfc3ed204d1" (UID: "90cba922-3d57-4b1f-9492-9cfc3ed204d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:34.024849 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90cba922-3d57-4b1f-9492-9cfc3ed204d1" (UID: "90cba922-3d57-4b1f-9492-9cfc3ed204d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:34.025038 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.024931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90cba922-3d57-4b1f-9492-9cfc3ed204d1" (UID: "90cba922-3d57-4b1f-9492-9cfc3ed204d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:34.026743 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.026717    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cba922-3d57-4b1f-9492-9cfc3ed204d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90cba922-3d57-4b1f-9492-9cfc3ed204d1" (UID: "90cba922-3d57-4b1f-9492-9cfc3ed204d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:34.026743 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.026718    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-api-access-744wq" (OuterVolumeSpecName: "kube-api-access-744wq") pod "90cba922-3d57-4b1f-9492-9cfc3ed204d1" (UID: "90cba922-3d57-4b1f-9492-9cfc3ed204d1"). InnerVolumeSpecName "kube-api-access-744wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:34.125332 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.125301    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90cba922-3d57-4b1f-9492-9cfc3ed204d1-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:34.125332 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.125334    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:34.125526 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.125349    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-744wq\" (UniqueName: \"kubernetes.io/projected/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-api-access-744wq\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:34.125526 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.125361    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:34.125526 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.125370    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90cba922-3d57-4b1f-9492-9cfc3ed204d1-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:34.891796 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.891756    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: i/o timeout"
Sep 11 16:33:34.907552 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.907515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk" event={"ID":"90cba922-3d57-4b1f-9492-9cfc3ed204d1","Type":"ContainerDied","Data":"e00d899d0aed82f50699244ae11c66d8bb58f2ac9a776d25dab51f2e9d68ceba"}
Sep 11 16:33:34.907713 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.907572    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"
Sep 11 16:33:34.907713 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.907589    2578 scope.go:117] "RemoveContainer" containerID="021908dbed5a383817a2d5eaf37a36e0d93169fd0c0c915d57610b06e8c91e8a"
Sep 11 16:33:34.916330 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.916309    2578 scope.go:117] "RemoveContainer" containerID="0c14bd6636dc1881b3ed656dfa2647e85a365846af62271941edaf49f1770b19"
Sep 11 16:33:34.924623 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.924598    2578 scope.go:117] "RemoveContainer" containerID="08b080648de182564828576696b89c8e4ff04fe1ec5ff94049fb5a3a9ebf4532"
Sep 11 16:33:34.929609 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.929582    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"]
Sep 11 16:33:34.933244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:34.933212    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d91b1e-predictor-7b6554f67c-n9cmk"]
Sep 11 16:33:35.895985 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:35.895947    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" path="/var/lib/kubelet/pods/90cba922-3d57-4b1f-9492-9cfc3ed204d1/volumes"
Sep 11 16:33:40.717629 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:40.717601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t_fa54cfd9-b196-400e-9239-3682f7aae5ab/kserve-container/0.log"
Sep 11 16:33:40.888144 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:40.888109    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"]
Sep 11 16:33:40.888399 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:40.888362    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kserve-container" containerID="cri-o://ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f" gracePeriod=30
Sep 11 16:33:40.888520 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:40.888399    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kube-rbac-proxy" containerID="cri-o://ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d" gracePeriod=30
Sep 11 16:33:41.009085 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009036    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"]
Sep 11 16:33:41.009395 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009382    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="storage-initializer"
Sep 11 16:33:41.009445 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009397    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="storage-initializer"
Sep 11 16:33:41.009445 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009408    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db"
Sep 11 16:33:41.009445 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009418    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db"
Sep 11 16:33:41.009445 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009443    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container"
Sep 11 16:33:41.009588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009449    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container"
Sep 11 16:33:41.009588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009456    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kube-rbac-proxy"
Sep 11 16:33:41.009588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kube-rbac-proxy"
Sep 11 16:33:41.009588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009512    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kserve-container"
Sep 11 16:33:41.009588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009520    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec4be61-426c-4012-8017-4b3d0afc3250" containerName="model-chainer-raw-hpa-785db"
Sep 11 16:33:41.009588 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.009528    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cba922-3d57-4b1f-9492-9cfc3ed204d1" containerName="kube-rbac-proxy"
Sep 11 16:33:41.013115 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.013090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.015402 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.015370    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b033f-predictor-serving-cert\""
Sep 11 16:33:41.015719 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.015611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:33:41.015719 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.015634    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:33:41.015915 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.015898    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-b033f-kube-rbac-proxy-sar-config\""
Sep 11 16:33:41.020833 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.020806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"]
Sep 11 16:33:41.080766 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.080734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbtr\" (UniqueName: \"kubernetes.io/projected/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-api-access-jwbtr\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.080935 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.080774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.080935 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.080794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.080935 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.080870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.080935 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.080911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.145123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.145096    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:33:41.181660 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.181613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.181866 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.181669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.181866 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.181784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbtr\" (UniqueName: \"kubernetes.io/projected/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-api-access-jwbtr\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.181866 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.181831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.181866 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.181862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.182089 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:33:41.181977    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-serving-cert: secret "isvc-sklearn-scale-raw-b033f-predictor-serving-cert" not found
Sep 11 16:33:41.182089 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:33:41.182042    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls podName:4ad64faf-1456-4a6c-ac3e-c5fa1f070bec nodeName:}" failed. No retries permitted until 2026-09-11 16:33:41.682021555 +0000 UTC m=+966.299991045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls") pod "isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" (UID: "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec") : secret "isvc-sklearn-scale-raw-b033f-predictor-serving-cert" not found
Sep 11 16:33:41.182201 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.182170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.182495 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.182473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-cabundle-cert\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.182552 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.182534    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.191680 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.191650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbtr\" (UniqueName: \"kubernetes.io/projected/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-api-access-jwbtr\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.282436 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.282334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls\") pod \"fa54cfd9-b196-400e-9239-3682f7aae5ab\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") "
Sep 11 16:33:41.282436 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.282387    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7qnd\" (UniqueName: \"kubernetes.io/projected/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-api-access-k7qnd\") pod \"fa54cfd9-b196-400e-9239-3682f7aae5ab\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") "
Sep 11 16:33:41.282436 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.282415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-rbac-proxy-sar-config\") pod \"fa54cfd9-b196-400e-9239-3682f7aae5ab\" (UID: \"fa54cfd9-b196-400e-9239-3682f7aae5ab\") "
Sep 11 16:33:41.282809 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.282788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa54cfd9-b196-400e-9239-3682f7aae5ab" (UID: "fa54cfd9-b196-400e-9239-3682f7aae5ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:41.284570 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.284546    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa54cfd9-b196-400e-9239-3682f7aae5ab" (UID: "fa54cfd9-b196-400e-9239-3682f7aae5ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:41.284638 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.284618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-api-access-k7qnd" (OuterVolumeSpecName: "kube-api-access-k7qnd") pod "fa54cfd9-b196-400e-9239-3682f7aae5ab" (UID: "fa54cfd9-b196-400e-9239-3682f7aae5ab"). InnerVolumeSpecName "kube-api-access-k7qnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:41.383468 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.383427    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:41.383468 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.383468    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa54cfd9-b196-400e-9239-3682f7aae5ab-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:41.383633 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.383479    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qnd\" (UniqueName: \"kubernetes.io/projected/fa54cfd9-b196-400e-9239-3682f7aae5ab-kube-api-access-k7qnd\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:33:41.684958 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.684913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.687470 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.687440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls\") pod \"isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.926024 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.925983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:41.938038 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938000    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerID="ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d" exitCode=2
Sep 11 16:33:41.938038 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938038    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerID="ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f" exitCode=2
Sep 11 16:33:41.938213 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938139    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"
Sep 11 16:33:41.938213 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" event={"ID":"fa54cfd9-b196-400e-9239-3682f7aae5ab","Type":"ContainerDied","Data":"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"}
Sep 11 16:33:41.938213 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" event={"ID":"fa54cfd9-b196-400e-9239-3682f7aae5ab","Type":"ContainerDied","Data":"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"}
Sep 11 16:33:41.938365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t" event={"ID":"fa54cfd9-b196-400e-9239-3682f7aae5ab","Type":"ContainerDied","Data":"7be43d1467645a369fdf7c3dbab213cff973aaf594cb4b0884c9ccdbb3cfc905"}
Sep 11 16:33:41.938365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.938257    2578 scope.go:117] "RemoveContainer" containerID="ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"
Sep 11 16:33:41.952017 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.951983    2578 scope.go:117] "RemoveContainer" containerID="ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"
Sep 11 16:33:41.960827 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.960795    2578 scope.go:117] "RemoveContainer" containerID="ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"
Sep 11 16:33:41.961524 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:33:41.961346    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d\": container with ID starting with ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d not found: ID does not exist" containerID="ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"
Sep 11 16:33:41.961524 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.961382    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"} err="failed to get container status \"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d\": rpc error: code = NotFound desc = could not find container \"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d\": container with ID starting with ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d not found: ID does not exist"
Sep 11 16:33:41.961524 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.961404    2578 scope.go:117] "RemoveContainer" containerID="ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"
Sep 11 16:33:41.961954 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:33:41.961934    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f\": container with ID starting with ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f not found: ID does not exist" containerID="ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"
Sep 11 16:33:41.962066 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.961966    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"} err="failed to get container status \"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f\": rpc error: code = NotFound desc = could not find container \"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f\": container with ID starting with ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f not found: ID does not exist"
Sep 11 16:33:41.962066 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.961990    2578 scope.go:117] "RemoveContainer" containerID="ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"
Sep 11 16:33:41.962417 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.962390    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d"} err="failed to get container status \"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d\": rpc error: code = NotFound desc = could not find container \"ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d\": container with ID starting with ae77ae4f83ef59458131edb03465b19f99bfa7e785db3706898df96c64c8382d not found: ID does not exist"
Sep 11 16:33:41.962474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.962420    2578 scope.go:117] "RemoveContainer" containerID="ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"
Sep 11 16:33:41.962735 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.962714    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f"} err="failed to get container status \"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f\": rpc error: code = NotFound desc = could not find container \"ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f\": container with ID starting with ad7679dd42d80348fc413df97512e531d798a06bf57bfc121dd9d35c9297ad9f not found: ID does not exist"
Sep 11 16:33:41.963346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.963328    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"]
Sep 11 16:33:41.967231 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:41.967202    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-ecc5e-predictor-54496d8b59-zkb5t"]
Sep 11 16:33:42.203487 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:42.203448    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"]
Sep 11 16:33:42.205794 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:33:42.205762    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad64faf_1456_4a6c_ac3e_c5fa1f070bec.slice/crio-067e43a31a908473d83d614e1356ae57794a08192b233c84c6537ebf9f8d0035 WatchSource:0}: Error finding container 067e43a31a908473d83d614e1356ae57794a08192b233c84c6537ebf9f8d0035: Status 404 returned error can't find the container with id 067e43a31a908473d83d614e1356ae57794a08192b233c84c6537ebf9f8d0035
Sep 11 16:33:42.944636 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:42.944593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerStarted","Data":"5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f"}
Sep 11 16:33:42.945003 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:42.944681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerStarted","Data":"067e43a31a908473d83d614e1356ae57794a08192b233c84c6537ebf9f8d0035"}
Sep 11 16:33:43.895551 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:43.895520    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" path="/var/lib/kubelet/pods/fa54cfd9-b196-400e-9239-3682f7aae5ab/volumes"
Sep 11 16:33:46.962994 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:46.962956    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerID="5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f" exitCode=0
Sep 11 16:33:46.963442 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:46.963005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerDied","Data":"5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f"}
Sep 11 16:33:47.969209 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:47.969169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerStarted","Data":"46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e"}
Sep 11 16:33:47.969673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:47.969279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerStarted","Data":"b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330"}
Sep 11 16:33:47.969673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:47.969504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:47.990639 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:47.990562    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podStartSLOduration=4.056127483 podStartE2EDuration="7.990543139s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934415656s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:47.98858015 +0000 UTC m=+972.606549663" watchObservedRunningTime="2026-09-11 16:33:47.990543139 +0000 UTC m=+972.608512640"
Sep 11 16:33:48.974057 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:48.974015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:48.975331 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:48.975300    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:33:49.192998 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.192967    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"]
Sep 11 16:33:49.193373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.193356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kube-rbac-proxy"
Sep 11 16:33:49.193430 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.193378    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kube-rbac-proxy"
Sep 11 16:33:49.193430 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.193389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kserve-container"
Sep 11 16:33:49.193430 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.193395    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kserve-container"
Sep 11 16:33:49.193548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.193467    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kube-rbac-proxy"
Sep 11 16:33:49.193548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.193477    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa54cfd9-b196-400e-9239-3682f7aae5ab" containerName="kserve-container"
Sep 11 16:33:49.196665 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.196641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.199241 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.199215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b6fc7-predictor-serving-cert\""
Sep 11 16:33:49.199375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.199279    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b6fc7-kube-rbac-proxy-sar-config\""
Sep 11 16:33:49.208262 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.208231    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"]
Sep 11 16:33:49.347258 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.347142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.347258 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.347225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4tc\" (UniqueName: \"kubernetes.io/projected/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-api-access-vd4tc\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.347516 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.347274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-cabundle-cert\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.347516 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.347353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.347516 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.347399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea924be6-2ba1-400c-b3be-16c078e458e5-kserve-provision-location\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.448485 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.448445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4tc\" (UniqueName: \"kubernetes.io/projected/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-api-access-vd4tc\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.448485 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.448484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-cabundle-cert\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.448752 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.448511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.448752 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.448535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea924be6-2ba1-400c-b3be-16c078e458e5-kserve-provision-location\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.448752 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.448586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.448752 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:33:49.448715    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-serving-cert: secret "raw-sklearn-b6fc7-predictor-serving-cert" not found
Sep 11 16:33:49.448954 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:33:49.448775    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls podName:ea924be6-2ba1-400c-b3be-16c078e458e5 nodeName:}" failed. No retries permitted until 2026-09-11 16:33:49.948755262 +0000 UTC m=+974.566724756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls") pod "raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" (UID: "ea924be6-2ba1-400c-b3be-16c078e458e5") : secret "raw-sklearn-b6fc7-predictor-serving-cert" not found
Sep 11 16:33:49.449017 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.449003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea924be6-2ba1-400c-b3be-16c078e458e5-kserve-provision-location\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.449273 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.449251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-cabundle-cert\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.449337 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.449260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.457979 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.457948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4tc\" (UniqueName: \"kubernetes.io/projected/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-api-access-vd4tc\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.953113 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.953068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.955842 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.955813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls\") pod \"raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") " pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:49.978244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:49.978199    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:33:50.109595 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:50.109554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:50.385961 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:50.385927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"]
Sep 11 16:33:50.389218 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:33:50.389186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea924be6_2ba1_400c_b3be_16c078e458e5.slice/crio-88034df6f4f4350308deba7bff1daefd1086808b8fb53cf6294145d9ef1ea022 WatchSource:0}: Error finding container 88034df6f4f4350308deba7bff1daefd1086808b8fb53cf6294145d9ef1ea022: Status 404 returned error can't find the container with id 88034df6f4f4350308deba7bff1daefd1086808b8fb53cf6294145d9ef1ea022
Sep 11 16:33:50.985193 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:50.985155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerStarted","Data":"3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717"}
Sep 11 16:33:50.985557 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:50.985237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerStarted","Data":"88034df6f4f4350308deba7bff1daefd1086808b8fb53cf6294145d9ef1ea022"}
Sep 11 16:33:54.982621 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:54.982594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:33:54.983139 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:54.983112    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:33:55.004210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:55.004171    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerID="3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717" exitCode=0
Sep 11 16:33:55.004365 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:55.004241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerDied","Data":"3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717"}
Sep 11 16:33:56.011372 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:56.011338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerStarted","Data":"ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0"}
Sep 11 16:33:56.011785 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:56.011402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerStarted","Data":"76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d"}
Sep 11 16:33:56.011785 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:56.011682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:56.011785 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:56.011728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:33:56.013196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:56.013165    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:33:56.031723 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:56.031665    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podStartSLOduration=3.073321514 podStartE2EDuration="7.031645807s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.958324293s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:56.029568825 +0000 UTC m=+980.647538339" watchObservedRunningTime="2026-09-11 16:33:56.031645807 +0000 UTC m=+980.649615308"
Sep 11 16:33:57.016415 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:33:57.016327    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:02.021375 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:02.021343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:34:02.021851 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:02.021824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:04.983262 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:04.983214    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:34:12.021922 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:12.021879    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:14.983700 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:14.983655    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:34:22.022763 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:22.022719    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:24.983488 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:24.983436    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:34:32.022729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:32.022690    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:34.983118 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:34.983069    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:34:42.021793 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:42.021748    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:44.983619 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:44.983581    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:34:52.022785 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:52.022738    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:34:54.983455 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:34:54.983412    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:35:02.022607 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:02.022564    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:35:04.983644 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:04.983597    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:35:12.021776 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:12.021736    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:35:14.983418 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:14.983380    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:35:22.023484 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:22.023447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:35:22.891257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:22.891219    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:35:29.323689 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:29.323607    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"]
Sep 11 16:35:29.324086 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:29.323842    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" containerID="cri-o://76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d" gracePeriod=30
Sep 11 16:35:29.324086 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:29.323890    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kube-rbac-proxy" containerID="cri-o://ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0" gracePeriod=30
Sep 11 16:35:30.423355 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:30.423323    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerID="ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0" exitCode=2
Sep 11 16:35:30.423741 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:30.423378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerDied","Data":"ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0"}
Sep 11 16:35:32.017227 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:32.017188    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 11 16:35:32.022587 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:32.022548    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 11 16:35:32.891691 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:32.891647    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:35:34.293939 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.293906    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:35:34.383351 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.383317    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd4tc\" (UniqueName: \"kubernetes.io/projected/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-api-access-vd4tc\") pod \"ea924be6-2ba1-400c-b3be-16c078e458e5\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") "
Sep 11 16:35:34.383634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.383393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea924be6-2ba1-400c-b3be-16c078e458e5-kserve-provision-location\") pod \"ea924be6-2ba1-400c-b3be-16c078e458e5\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") "
Sep 11 16:35:34.383634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.383423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-rbac-proxy-sar-config\") pod \"ea924be6-2ba1-400c-b3be-16c078e458e5\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") "
Sep 11 16:35:34.383634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.383521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls\") pod \"ea924be6-2ba1-400c-b3be-16c078e458e5\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") "
Sep 11 16:35:34.383634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.383574    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-cabundle-cert\") pod \"ea924be6-2ba1-400c-b3be-16c078e458e5\" (UID: \"ea924be6-2ba1-400c-b3be-16c078e458e5\") "
Sep 11 16:35:34.384114 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.384085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea924be6-2ba1-400c-b3be-16c078e458e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea924be6-2ba1-400c-b3be-16c078e458e5" (UID: "ea924be6-2ba1-400c-b3be-16c078e458e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:35:34.384218 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.384108    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea924be6-2ba1-400c-b3be-16c078e458e5" (UID: "ea924be6-2ba1-400c-b3be-16c078e458e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:35:34.384218 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.384122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea924be6-2ba1-400c-b3be-16c078e458e5" (UID: "ea924be6-2ba1-400c-b3be-16c078e458e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:35:34.385960 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.385931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea924be6-2ba1-400c-b3be-16c078e458e5" (UID: "ea924be6-2ba1-400c-b3be-16c078e458e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:35:34.386041 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.385958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-api-access-vd4tc" (OuterVolumeSpecName: "kube-api-access-vd4tc") pod "ea924be6-2ba1-400c-b3be-16c078e458e5" (UID: "ea924be6-2ba1-400c-b3be-16c078e458e5"). InnerVolumeSpecName "kube-api-access-vd4tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:35:34.441917 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.441880    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerID="76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d" exitCode=0
Sep 11 16:35:34.442130 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.441978    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"
Sep 11 16:35:34.442130 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.441977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerDied","Data":"76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d"}
Sep 11 16:35:34.442220 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.442179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6" event={"ID":"ea924be6-2ba1-400c-b3be-16c078e458e5","Type":"ContainerDied","Data":"88034df6f4f4350308deba7bff1daefd1086808b8fb53cf6294145d9ef1ea022"}
Sep 11 16:35:34.442260 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.442232    2578 scope.go:117] "RemoveContainer" containerID="ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0"
Sep 11 16:35:34.452861 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.452842    2578 scope.go:117] "RemoveContainer" containerID="76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d"
Sep 11 16:35:34.461182 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.461163    2578 scope.go:117] "RemoveContainer" containerID="3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717"
Sep 11 16:35:34.465847 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.465819    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"]
Sep 11 16:35:34.469974 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.469945    2578 scope.go:117] "RemoveContainer" containerID="ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0"
Sep 11 16:35:34.470800 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:35:34.470567    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0\": container with ID starting with ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0 not found: ID does not exist" containerID="ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0"
Sep 11 16:35:34.470800 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.470609    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0"} err="failed to get container status \"ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0\": rpc error: code = NotFound desc = could not find container \"ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0\": container with ID starting with ce5a7b2ef20ceb0db92add5ac43be6d8ad1a659e1cb5657e368526d46e97bee0 not found: ID does not exist"
Sep 11 16:35:34.470800 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.470636    2578 scope.go:117] "RemoveContainer" containerID="76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d"
Sep 11 16:35:34.471373 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:35:34.471006    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d\": container with ID starting with 76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d not found: ID does not exist" containerID="76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d"
Sep 11 16:35:34.471373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.471031    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d"} err="failed to get container status \"76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d\": rpc error: code = NotFound desc = could not find container \"76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d\": container with ID starting with 76010ca773539b7d23f839cc4be9036cdbd156d6e1ffa0fb3d41ce44d558d45d not found: ID does not exist"
Sep 11 16:35:34.471373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.471072    2578 scope.go:117] "RemoveContainer" containerID="3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717"
Sep 11 16:35:34.471539 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:35:34.471395    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717\": container with ID starting with 3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717 not found: ID does not exist" containerID="3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717"
Sep 11 16:35:34.471539 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.471421    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717"} err="failed to get container status \"3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717\": rpc error: code = NotFound desc = could not find container \"3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717\": container with ID starting with 3f947ed8dd0588d24222cf802db4c61d0b960449febff1e0ae059742f376f717 not found: ID does not exist"
Sep 11 16:35:34.472548 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.472529    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b6fc7-predictor-746cdf8948-z5dc6"]
Sep 11 16:35:34.485018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.484985    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:35:34.485018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.485013    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4tc\" (UniqueName: \"kubernetes.io/projected/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-api-access-vd4tc\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:35:34.485018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.485023    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea924be6-2ba1-400c-b3be-16c078e458e5-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:35:34.485256 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.485035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea924be6-2ba1-400c-b3be-16c078e458e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:35:34.485256 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:34.485061    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea924be6-2ba1-400c-b3be-16c078e458e5-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:35:35.895651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:35.895618    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" path="/var/lib/kubelet/pods/ea924be6-2ba1-400c-b3be-16c078e458e5/volumes"
Sep 11 16:35:42.891848 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:42.891807    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:35:52.891778 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:35:52.891734    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:36:02.891200 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:02.891158    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:36:12.891912 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:12.891861    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:36:22.891235 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:22.891190    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:36:32.892254 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:32.892223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:36:41.088471 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:41.088435    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"]
Sep 11 16:36:41.088991 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:41.088704    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" containerID="cri-o://b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330" gracePeriod=30
Sep 11 16:36:41.088991 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:41.088761    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kube-rbac-proxy" containerID="cri-o://46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e" gracePeriod=30
Sep 11 16:36:41.720307 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:41.720270    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerID="46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e" exitCode=2
Sep 11 16:36:41.720490 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:41.720309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerDied","Data":"46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e"}
Sep 11 16:36:42.891578 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:42.891525    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 11 16:36:44.978942 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:44.978889    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 11 16:36:46.294058 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294019    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"]
Sep 11 16:36:46.294426 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294413    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container"
Sep 11 16:36:46.294426 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container"
Sep 11 16:36:46.294512 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294436    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kube-rbac-proxy"
Sep 11 16:36:46.294512 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294443    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kube-rbac-proxy"
Sep 11 16:36:46.294512 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294454    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="storage-initializer"
Sep 11 16:36:46.294512 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294459    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="storage-initializer"
Sep 11 16:36:46.294512 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294513    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kube-rbac-proxy"
Sep 11 16:36:46.294676 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.294521    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea924be6-2ba1-400c-b3be-16c078e458e5" containerName="kserve-container"
Sep 11 16:36:46.297607 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.297589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.300285 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.300264    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 11 16:36:46.300464 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.300348    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 11 16:36:46.312104 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.312072    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"]
Sep 11 16:36:46.378918 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.378880    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"]
Sep 11 16:36:46.382246 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.382230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.385242 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.385221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 11 16:36:46.393189 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.393150    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"]
Sep 11 16:36:46.450794 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.450757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/945749d5-dc2c-49bc-8616-7828a96936e1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.450794 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.450797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/945749d5-dc2c-49bc-8616-7828a96936e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.451010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.450823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2gb\" (UniqueName: \"kubernetes.io/projected/945749d5-dc2c-49bc-8616-7828a96936e1-kube-api-access-4x2gb\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.451010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.450923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.451010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.450958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.551440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.551440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a24bcd-85f8-4ed0-9bb7-5128780de711-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.551440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.551732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.551732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.551732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbgw\" (UniqueName: \"kubernetes.io/projected/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-api-access-2fbgw\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.551732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/945749d5-dc2c-49bc-8616-7828a96936e1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.551732 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/945749d5-dc2c-49bc-8616-7828a96936e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.552314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2gb\" (UniqueName: \"kubernetes.io/projected/945749d5-dc2c-49bc-8616-7828a96936e1-kube-api-access-4x2gb\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.552314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.551780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a24bcd-85f8-4ed0-9bb7-5128780de711-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.552314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.552076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/945749d5-dc2c-49bc-8616-7828a96936e1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.552314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.552163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.552314 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.552194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.554332 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.554316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/945749d5-dc2c-49bc-8616-7828a96936e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.561011 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.560983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2gb\" (UniqueName: \"kubernetes.io/projected/945749d5-dc2c-49bc-8616-7828a96936e1-kube-api-access-4x2gb\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qdxc6\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.607543 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.607498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:46.652982 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.652949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a24bcd-85f8-4ed0-9bb7-5128780de711-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.653215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.652994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.653215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.653029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.653215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.653079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbgw\" (UniqueName: \"kubernetes.io/projected/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-api-access-2fbgw\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.653215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.653134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a24bcd-85f8-4ed0-9bb7-5128780de711-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.653660 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.653623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a24bcd-85f8-4ed0-9bb7-5128780de711-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.654330 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.654282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.654441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.654347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.657441 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.657186    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a24bcd-85f8-4ed0-9bb7-5128780de711-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.663031 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.662978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbgw\" (UniqueName: \"kubernetes.io/projected/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-api-access-2fbgw\") pod \"isvc-canary-create-predictor-6474b875b9-4dfjh\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.693002 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.692971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:46.881601 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.881570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"]
Sep 11 16:36:46.883367 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:36:46.883338    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945749d5_dc2c_49bc_8616_7828a96936e1.slice/crio-e4f461ca00b76d76213b8a8c284840cf14860783ba3bd3d3cbb5c173704dd0ec WatchSource:0}: Error finding container e4f461ca00b76d76213b8a8c284840cf14860783ba3bd3d3cbb5c173704dd0ec: Status 404 returned error can't find the container with id e4f461ca00b76d76213b8a8c284840cf14860783ba3bd3d3cbb5c173704dd0ec
Sep 11 16:36:46.969349 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:46.969318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"]
Sep 11 16:36:46.972175 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:36:46.972151    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a24bcd_85f8_4ed0_9bb7_5128780de711.slice/crio-a0bb133d5455fd6470b07622d8eec124ca9c5479a560fcc689b6de7255c82e37 WatchSource:0}: Error finding container a0bb133d5455fd6470b07622d8eec124ca9c5479a560fcc689b6de7255c82e37: Status 404 returned error can't find the container with id a0bb133d5455fd6470b07622d8eec124ca9c5479a560fcc689b6de7255c82e37
Sep 11 16:36:47.746904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:47.746862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerStarted","Data":"1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d"}
Sep 11 16:36:47.747394 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:47.746945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerStarted","Data":"e4f461ca00b76d76213b8a8c284840cf14860783ba3bd3d3cbb5c173704dd0ec"}
Sep 11 16:36:47.748556 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:47.748531    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerStarted","Data":"c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5"}
Sep 11 16:36:47.748653 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:47.748589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerStarted","Data":"a0bb133d5455fd6470b07622d8eec124ca9c5479a560fcc689b6de7255c82e37"}
Sep 11 16:36:49.979361 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:49.979315    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 11 16:36:51.139525 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.139500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:36:51.191716 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.191676    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-cabundle-cert\") pod \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") "
Sep 11 16:36:51.191716 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.191722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls\") pod \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") "
Sep 11 16:36:51.191973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.191743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwbtr\" (UniqueName: \"kubernetes.io/projected/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-api-access-jwbtr\") pod \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") "
Sep 11 16:36:51.191973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.191774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kserve-provision-location\") pod \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") "
Sep 11 16:36:51.191973 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.191846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-rbac-proxy-sar-config\") pod \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\" (UID: \"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec\") "
Sep 11 16:36:51.192191 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.192140    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" (UID: "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:51.192191 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.192151    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" (UID: "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:51.192270 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.192214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" (UID: "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:36:51.194010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.193990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-api-access-jwbtr" (OuterVolumeSpecName: "kube-api-access-jwbtr") pod "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" (UID: "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec"). InnerVolumeSpecName "kube-api-access-jwbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:36:51.194087 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.194021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" (UID: "4ad64faf-1456-4a6c-ac3e-c5fa1f070bec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:36:51.292818 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.292784    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:36:51.292818 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.292812    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:36:51.292818 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.292822    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbtr\" (UniqueName: \"kubernetes.io/projected/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-api-access-jwbtr\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:36:51.293020 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.292832    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:36:51.293020 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.292841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:36:51.769099 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.769063    2578 generic.go:356] "Generic (PLEG): container finished" podID="945749d5-dc2c-49bc-8616-7828a96936e1" containerID="1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d" exitCode=0
Sep 11 16:36:51.769292 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.769120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerDied","Data":"1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d"}
Sep 11 16:36:51.770582 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.770563    2578 generic.go:356] "Generic (PLEG): container finished" podID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerID="c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5" exitCode=0
Sep 11 16:36:51.770673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.770631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerDied","Data":"c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5"}
Sep 11 16:36:51.772607 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.772587    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerID="b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330" exitCode=0
Sep 11 16:36:51.772705 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.772612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerDied","Data":"b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330"}
Sep 11 16:36:51.772705 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.772644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn" event={"ID":"4ad64faf-1456-4a6c-ac3e-c5fa1f070bec","Type":"ContainerDied","Data":"067e43a31a908473d83d614e1356ae57794a08192b233c84c6537ebf9f8d0035"}
Sep 11 16:36:51.772705 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.772658    2578 scope.go:117] "RemoveContainer" containerID="46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e"
Sep 11 16:36:51.772863 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.772643    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"
Sep 11 16:36:51.784076 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.784036    2578 scope.go:117] "RemoveContainer" containerID="b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330"
Sep 11 16:36:51.807028 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.806999    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"]
Sep 11 16:36:51.807502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.807485    2578 scope.go:117] "RemoveContainer" containerID="5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f"
Sep 11 16:36:51.812428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.812405    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-b033f-predictor-689959ddc8-lzvvn"]
Sep 11 16:36:51.823233 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.823205    2578 scope.go:117] "RemoveContainer" containerID="46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e"
Sep 11 16:36:51.823794 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:36:51.823623    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e\": container with ID starting with 46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e not found: ID does not exist" containerID="46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e"
Sep 11 16:36:51.823794 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.823660    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e"} err="failed to get container status \"46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e\": rpc error: code = NotFound desc = could not find container \"46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e\": container with ID starting with 46ff91470b45c1120cdbe2d34148f7ce65b8ce976ac5f207c0be2b25ebab135e not found: ID does not exist"
Sep 11 16:36:51.823794 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.823687    2578 scope.go:117] "RemoveContainer" containerID="b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330"
Sep 11 16:36:51.824110 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:36:51.824081    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330\": container with ID starting with b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330 not found: ID does not exist" containerID="b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330"
Sep 11 16:36:51.824202 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.824121    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330"} err="failed to get container status \"b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330\": rpc error: code = NotFound desc = could not find container \"b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330\": container with ID starting with b065d822c3ff455b542e73cd57116a8ed35d32af2386c02dd66163c453b34330 not found: ID does not exist"
Sep 11 16:36:51.824202 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.824141    2578 scope.go:117] "RemoveContainer" containerID="5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f"
Sep 11 16:36:51.824605 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:36:51.824576    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f\": container with ID starting with 5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f not found: ID does not exist" containerID="5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f"
Sep 11 16:36:51.824716 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.824615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f"} err="failed to get container status \"5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f\": rpc error: code = NotFound desc = could not find container \"5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f\": container with ID starting with 5d1680f8d7c5fa3a1122c31f5386926385b2b34cb72efb64002b5b5a8d310b1f not found: ID does not exist"
Sep 11 16:36:51.896820 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:51.896790    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" path="/var/lib/kubelet/pods/4ad64faf-1456-4a6c-ac3e-c5fa1f070bec/volumes"
Sep 11 16:36:52.779479 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.779442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerStarted","Data":"24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5"}
Sep 11 16:36:52.779946 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.779571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerStarted","Data":"5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4"}
Sep 11 16:36:52.780095 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.780073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:52.780167 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.780133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:52.781331 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.781303    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:36:52.782188 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.782164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerStarted","Data":"c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab"}
Sep 11 16:36:52.782305 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.782225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerStarted","Data":"39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b"}
Sep 11 16:36:52.782481 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.782459    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:52.801062 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.800988    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podStartSLOduration=2.899640752 podStartE2EDuration="6.800970508s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.901329756s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:52.799361044 +0000 UTC m=+1157.417330547" watchObservedRunningTime="2026-09-11 16:36:52.800970508 +0000 UTC m=+1157.418940009"
Sep 11 16:36:52.820402 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:52.820346    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podStartSLOduration=3.066190257 podStartE2EDuration="6.820329345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.754139088s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:52.819133828 +0000 UTC m=+1157.437103332" watchObservedRunningTime="2026-09-11 16:36:52.820329345 +0000 UTC m=+1157.438298846"
Sep 11 16:36:53.788482 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:53.788441    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:36:53.788957 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:53.788573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:53.789971 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:53.789943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:36:54.793252 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:54.793209    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:36:58.793087 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:58.792984    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:36:58.793527 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:58.793491    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:36:59.798208 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:59.798177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:36:59.798782 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:36:59.798757    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:08.793791 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:08.793754    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:37:09.799673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:09.799637    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:18.794228 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:18.794189    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:37:19.799727 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:19.799685    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:28.793423 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:28.793379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:37:29.799291 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:29.799246    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:38.794353 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:38.794308    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:37:39.799291 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:39.799250    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:48.794222 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:48.794177    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:37:49.799766 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:49.799723    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:58.794409 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:58.794361    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:37:59.799346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:59.799307    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 11 16:37:59.891227 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:37:59.891191    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:38:09.800133 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:09.800102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:38:09.895839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:09.895813    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:38:10.040515 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:10.040476    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"]
Sep 11 16:38:10.113830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:10.113780    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container" containerID="cri-o://39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b" gracePeriod=30
Sep 11 16:38:10.113830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:10.113823    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kube-rbac-proxy" containerID="cri-o://c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab" gracePeriod=30
Sep 11 16:38:11.120886 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:11.120854    2578 generic.go:356] "Generic (PLEG): container finished" podID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerID="c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab" exitCode=2
Sep 11 16:38:11.121300 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:11.120925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerDied","Data":"c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab"}
Sep 11 16:38:14.794348 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:14.794304    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 11 16:38:15.058642 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.058616    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:38:15.139718 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.139679    2578 generic.go:356] "Generic (PLEG): container finished" podID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerID="39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b" exitCode=0
Sep 11 16:38:15.139890 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.139728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerDied","Data":"39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b"}
Sep 11 16:38:15.139890 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.139747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"
Sep 11 16:38:15.139890 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.139796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh" event={"ID":"93a24bcd-85f8-4ed0-9bb7-5128780de711","Type":"ContainerDied","Data":"a0bb133d5455fd6470b07622d8eec124ca9c5479a560fcc689b6de7255c82e37"}
Sep 11 16:38:15.139890 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.139825    2578 scope.go:117] "RemoveContainer" containerID="c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab"
Sep 11 16:38:15.148131 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.148109    2578 scope.go:117] "RemoveContainer" containerID="39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b"
Sep 11 16:38:15.155811 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.155791    2578 scope.go:117] "RemoveContainer" containerID="c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5"
Sep 11 16:38:15.163651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.163628    2578 scope.go:117] "RemoveContainer" containerID="c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab"
Sep 11 16:38:15.163976 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:15.163954    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab\": container with ID starting with c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab not found: ID does not exist" containerID="c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab"
Sep 11 16:38:15.164026 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.163990    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab"} err="failed to get container status \"c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab\": rpc error: code = NotFound desc = could not find container \"c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab\": container with ID starting with c0a4d3520d99c365126ceb51f1e10651508cf1832b780d92064e435e774be7ab not found: ID does not exist"
Sep 11 16:38:15.164026 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.164010    2578 scope.go:117] "RemoveContainer" containerID="39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b"
Sep 11 16:38:15.164310 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:15.164295    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b\": container with ID starting with 39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b not found: ID does not exist" containerID="39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b"
Sep 11 16:38:15.164374 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.164312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b"} err="failed to get container status \"39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b\": rpc error: code = NotFound desc = could not find container \"39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b\": container with ID starting with 39f6f5676b039456eac7b7c674f2020e485548be9bae41e542ac25435a553b2b not found: ID does not exist"
Sep 11 16:38:15.164374 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.164326    2578 scope.go:117] "RemoveContainer" containerID="c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5"
Sep 11 16:38:15.164593 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:15.164575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5\": container with ID starting with c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5 not found: ID does not exist" containerID="c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5"
Sep 11 16:38:15.164634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.164599    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5"} err="failed to get container status \"c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5\": rpc error: code = NotFound desc = could not find container \"c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5\": container with ID starting with c26a4ad9d257147d5c8daf09d02de379d7154ca93059dbf395d2dca1c5f675e5 not found: ID does not exist"
Sep 11 16:38:15.174716 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.174689    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-rbac-proxy-sar-config\") pod \"93a24bcd-85f8-4ed0-9bb7-5128780de711\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") "
Sep 11 16:38:15.174862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.174748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a24bcd-85f8-4ed0-9bb7-5128780de711-proxy-tls\") pod \"93a24bcd-85f8-4ed0-9bb7-5128780de711\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") "
Sep 11 16:38:15.174862 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.174776    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-cabundle-cert\") pod \"93a24bcd-85f8-4ed0-9bb7-5128780de711\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") "
Sep 11 16:38:15.174976 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.174938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fbgw\" (UniqueName: \"kubernetes.io/projected/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-api-access-2fbgw\") pod \"93a24bcd-85f8-4ed0-9bb7-5128780de711\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") "
Sep 11 16:38:15.175035 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.175012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a24bcd-85f8-4ed0-9bb7-5128780de711-kserve-provision-location\") pod \"93a24bcd-85f8-4ed0-9bb7-5128780de711\" (UID: \"93a24bcd-85f8-4ed0-9bb7-5128780de711\") "
Sep 11 16:38:15.175148 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.175125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93a24bcd-85f8-4ed0-9bb7-5128780de711" (UID: "93a24bcd-85f8-4ed0-9bb7-5128780de711"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:15.175200 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.175129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93a24bcd-85f8-4ed0-9bb7-5128780de711" (UID: "93a24bcd-85f8-4ed0-9bb7-5128780de711"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:15.175305 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.175282    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:15.175358 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.175309    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a24bcd-85f8-4ed0-9bb7-5128780de711-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:15.175392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.175363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a24bcd-85f8-4ed0-9bb7-5128780de711-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93a24bcd-85f8-4ed0-9bb7-5128780de711" (UID: "93a24bcd-85f8-4ed0-9bb7-5128780de711"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:15.177165 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.177143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a24bcd-85f8-4ed0-9bb7-5128780de711-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93a24bcd-85f8-4ed0-9bb7-5128780de711" (UID: "93a24bcd-85f8-4ed0-9bb7-5128780de711"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:15.177165 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.177145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-api-access-2fbgw" (OuterVolumeSpecName: "kube-api-access-2fbgw") pod "93a24bcd-85f8-4ed0-9bb7-5128780de711" (UID: "93a24bcd-85f8-4ed0-9bb7-5128780de711"). InnerVolumeSpecName "kube-api-access-2fbgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:15.275947 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.275896    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a24bcd-85f8-4ed0-9bb7-5128780de711-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:15.275947 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.275939    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93a24bcd-85f8-4ed0-9bb7-5128780de711-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:15.275947 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.275949    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fbgw\" (UniqueName: \"kubernetes.io/projected/93a24bcd-85f8-4ed0-9bb7-5128780de711-kube-api-access-2fbgw\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:15.461318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.461276    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"]
Sep 11 16:38:15.463932 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.463900    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-4dfjh"]
Sep 11 16:38:15.894901 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:15.894862    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" path="/var/lib/kubelet/pods/93a24bcd-85f8-4ed0-9bb7-5128780de711/volumes"
Sep 11 16:38:16.366330 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:16.366299    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"]
Sep 11 16:38:16.366627 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:16.366598    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" containerID="cri-o://5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4" gracePeriod=30
Sep 11 16:38:16.366740 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:16.366624    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kube-rbac-proxy" containerID="cri-o://24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5" gracePeriod=30
Sep 11 16:38:17.151536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:17.151502    2578 generic.go:356] "Generic (PLEG): container finished" podID="945749d5-dc2c-49bc-8616-7828a96936e1" containerID="24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5" exitCode=2
Sep 11 16:38:17.151934 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:17.151581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerDied","Data":"24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5"}
Sep 11 16:38:18.789010 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:18.788965    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 11 16:38:19.892086 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:19.892025    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 11 16:38:21.528078 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.528028    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:38:21.562805 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.562732    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"]
Sep 11 16:38:21.563073 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563060    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563075    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563084    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="storage-initializer"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563090    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="storage-initializer"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563100    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="storage-initializer"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563105    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="storage-initializer"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563119    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563123 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563125    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563130    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563136    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563147    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563152    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563158    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563163    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563172    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="storage-initializer"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563176    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="storage-initializer"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563221    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563228    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kserve-container"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563239    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad64faf-1456-4a6c-ac3e-c5fa1f070bec" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563244    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kube-rbac-proxy"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563250    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a24bcd-85f8-4ed0-9bb7-5128780de711" containerName="kserve-container"
Sep 11 16:38:21.563392 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.563255    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" containerName="kserve-container"
Sep 11 16:38:21.566342 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.566319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.568770 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.568750    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 11 16:38:21.569146 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.569128    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 11 16:38:21.575722 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.575697    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"]
Sep 11 16:38:21.623685 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.623644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/945749d5-dc2c-49bc-8616-7828a96936e1-proxy-tls\") pod \"945749d5-dc2c-49bc-8616-7828a96936e1\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") "
Sep 11 16:38:21.623923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.623755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-cabundle-cert\") pod \"945749d5-dc2c-49bc-8616-7828a96936e1\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") "
Sep 11 16:38:21.623923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.623801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/945749d5-dc2c-49bc-8616-7828a96936e1-kserve-provision-location\") pod \"945749d5-dc2c-49bc-8616-7828a96936e1\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") "
Sep 11 16:38:21.623923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.623826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x2gb\" (UniqueName: \"kubernetes.io/projected/945749d5-dc2c-49bc-8616-7828a96936e1-kube-api-access-4x2gb\") pod \"945749d5-dc2c-49bc-8616-7828a96936e1\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") "
Sep 11 16:38:21.623923 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.623852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-kube-rbac-proxy-sar-config\") pod \"945749d5-dc2c-49bc-8616-7828a96936e1\" (UID: \"945749d5-dc2c-49bc-8616-7828a96936e1\") "
Sep 11 16:38:21.624187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.624187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.624187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.624187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a985876-0fcf-4732-9855-00ce27a55d13-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.624187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624154    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945749d5-dc2c-49bc-8616-7828a96936e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "945749d5-dc2c-49bc-8616-7828a96936e1" (UID: "945749d5-dc2c-49bc-8616-7828a96936e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:21.624442 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntl9m\" (UniqueName: \"kubernetes.io/projected/3a985876-0fcf-4732-9855-00ce27a55d13-kube-api-access-ntl9m\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.624442 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "945749d5-dc2c-49bc-8616-7828a96936e1" (UID: "945749d5-dc2c-49bc-8616-7828a96936e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:21.624798 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624765    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/945749d5-dc2c-49bc-8616-7828a96936e1-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.624798 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624791    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.624953 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.624910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "945749d5-dc2c-49bc-8616-7828a96936e1" (UID: "945749d5-dc2c-49bc-8616-7828a96936e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:21.626323 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.626305    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945749d5-dc2c-49bc-8616-7828a96936e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "945749d5-dc2c-49bc-8616-7828a96936e1" (UID: "945749d5-dc2c-49bc-8616-7828a96936e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:21.626393 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.626374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945749d5-dc2c-49bc-8616-7828a96936e1-kube-api-access-4x2gb" (OuterVolumeSpecName: "kube-api-access-4x2gb") pod "945749d5-dc2c-49bc-8616-7828a96936e1" (UID: "945749d5-dc2c-49bc-8616-7828a96936e1"). InnerVolumeSpecName "kube-api-access-4x2gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:21.726199 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntl9m\" (UniqueName: \"kubernetes.io/projected/3a985876-0fcf-4732-9855-00ce27a55d13-kube-api-access-ntl9m\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.726373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.726373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.726373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.726373 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a985876-0fcf-4732-9855-00ce27a55d13-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.726583 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726391    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/945749d5-dc2c-49bc-8616-7828a96936e1-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.726583 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2gb\" (UniqueName: \"kubernetes.io/projected/945749d5-dc2c-49bc-8616-7828a96936e1-kube-api-access-4x2gb\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.726583 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726418    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/945749d5-dc2c-49bc-8616-7828a96936e1-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:38:21.726583 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:21.726431    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 11 16:38:21.726583 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:21.726518    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls podName:3a985876-0fcf-4732-9855-00ce27a55d13 nodeName:}" failed. No retries permitted until 2026-09-11 16:38:22.226496787 +0000 UTC m=+1246.844466290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls") pod "isvc-canary-promote-predictor-7cf78f884f-pfszw" (UID: "3a985876-0fcf-4732-9855-00ce27a55d13") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 11 16:38:21.726840 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a985876-0fcf-4732-9855-00ce27a55d13-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.726992 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.726972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.727134 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.727115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:21.735039 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:21.735013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntl9m\" (UniqueName: \"kubernetes.io/projected/3a985876-0fcf-4732-9855-00ce27a55d13-kube-api-access-ntl9m\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:22.173718 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.173630    2578 generic.go:356] "Generic (PLEG): container finished" podID="945749d5-dc2c-49bc-8616-7828a96936e1" containerID="5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4" exitCode=0
Sep 11 16:38:22.173718 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.173689    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"
Sep 11 16:38:22.173718 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.173706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerDied","Data":"5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4"}
Sep 11 16:38:22.173945 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.173768    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6" event={"ID":"945749d5-dc2c-49bc-8616-7828a96936e1","Type":"ContainerDied","Data":"e4f461ca00b76d76213b8a8c284840cf14860783ba3bd3d3cbb5c173704dd0ec"}
Sep 11 16:38:22.173945 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.173794    2578 scope.go:117] "RemoveContainer" containerID="24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5"
Sep 11 16:38:22.184235 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.184218    2578 scope.go:117] "RemoveContainer" containerID="5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4"
Sep 11 16:38:22.192289 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.192272    2578 scope.go:117] "RemoveContainer" containerID="1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d"
Sep 11 16:38:22.193424 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.193403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"]
Sep 11 16:38:22.198248 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.198228    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qdxc6"]
Sep 11 16:38:22.200842 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.200826    2578 scope.go:117] "RemoveContainer" containerID="24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5"
Sep 11 16:38:22.201131 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:22.201114    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5\": container with ID starting with 24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5 not found: ID does not exist" containerID="24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5"
Sep 11 16:38:22.201183 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.201144    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5"} err="failed to get container status \"24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5\": rpc error: code = NotFound desc = could not find container \"24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5\": container with ID starting with 24e5c9d5572d6b74d2370dd8b3d78c391e083560002ff7157887a81d49faf5d5 not found: ID does not exist"
Sep 11 16:38:22.201183 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.201166    2578 scope.go:117] "RemoveContainer" containerID="5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4"
Sep 11 16:38:22.201442 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:22.201424    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4\": container with ID starting with 5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4 not found: ID does not exist" containerID="5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4"
Sep 11 16:38:22.201493 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.201452    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4"} err="failed to get container status \"5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4\": rpc error: code = NotFound desc = could not find container \"5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4\": container with ID starting with 5802e20973d4648137c6f0cbb589127071dd0d66821671e5fcea2ce1fe9237b4 not found: ID does not exist"
Sep 11 16:38:22.201493 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.201467    2578 scope.go:117] "RemoveContainer" containerID="1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d"
Sep 11 16:38:22.201692 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:38:22.201677    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d\": container with ID starting with 1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d not found: ID does not exist" containerID="1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d"
Sep 11 16:38:22.201774 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.201696    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d"} err="failed to get container status \"1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d\": rpc error: code = NotFound desc = could not find container \"1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d\": container with ID starting with 1d014fd9acee99adbb2c6e0af209afcb4ac03f24e6cd6bcf84136ecc22ad822d not found: ID does not exist"
Sep 11 16:38:22.231783 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.231752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:22.234294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.234264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-pfszw\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:22.476641 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.476545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:22.750270 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:22.750242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"]
Sep 11 16:38:22.752834 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:38:22.752806    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a985876_0fcf_4732_9855_00ce27a55d13.slice/crio-56563a3cce524cf987795783c06cae1f55adb42034c86b9ab746c323e9fd097a WatchSource:0}: Error finding container 56563a3cce524cf987795783c06cae1f55adb42034c86b9ab746c323e9fd097a: Status 404 returned error can't find the container with id 56563a3cce524cf987795783c06cae1f55adb42034c86b9ab746c323e9fd097a
Sep 11 16:38:23.180978 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:23.180931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerStarted","Data":"10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f"}
Sep 11 16:38:23.181178 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:23.181015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerStarted","Data":"56563a3cce524cf987795783c06cae1f55adb42034c86b9ab746c323e9fd097a"}
Sep 11 16:38:23.895260 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:23.895223    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="945749d5-dc2c-49bc-8616-7828a96936e1" path="/var/lib/kubelet/pods/945749d5-dc2c-49bc-8616-7828a96936e1/volumes"
Sep 11 16:38:27.200713 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:27.200670    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a985876-0fcf-4732-9855-00ce27a55d13" containerID="10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f" exitCode=0
Sep 11 16:38:27.200713 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:27.200710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerDied","Data":"10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f"}
Sep 11 16:38:28.207728 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:28.207694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerStarted","Data":"4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf"}
Sep 11 16:38:28.208098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:28.207756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerStarted","Data":"9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3"}
Sep 11 16:38:28.208098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:28.208013    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:28.208250 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:28.208177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:28.209450 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:28.209425    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:38:28.230668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:28.230613    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podStartSLOduration=3.297713438 podStartE2EDuration="7.230597352s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.932883914s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:38:28.228766367 +0000 UTC m=+1252.846735905" watchObservedRunningTime="2026-09-11 16:38:28.230597352 +0000 UTC m=+1252.848566852"
Sep 11 16:38:29.212066 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:29.212002    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:38:34.217023 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:34.216992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:38:34.217718 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:34.217692    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:38:44.218178 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:44.218136    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:38:54.218490 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:38:54.218449    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:39:04.217686 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:04.217648    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:39:14.218338 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:14.218296    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:39:24.217599 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:24.217553    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:39:34.218195 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:34.218149    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 11 16:39:43.850239 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:43.850206    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"]
Sep 11 16:39:43.850750 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:43.850450    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container" containerID="cri-o://9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3" gracePeriod=30
Sep 11 16:39:43.850750 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:43.850544    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kube-rbac-proxy" containerID="cri-o://4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf" gracePeriod=30
Sep 11 16:39:43.851830 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:43.851770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:39:44.212776 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:44.212728    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 11 16:39:44.528834 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:44.528748    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a985876-0fcf-4732-9855-00ce27a55d13" containerID="4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf" exitCode=2
Sep 11 16:39:44.528834 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:44.528819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerDied","Data":"4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf"}
Sep 11 16:39:48.899990 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:48.899962    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:39:49.013977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.013874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-kube-rbac-proxy-sar-config\") pod \"3a985876-0fcf-4732-9855-00ce27a55d13\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") "
Sep 11 16:39:49.013977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.013931    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls\") pod \"3a985876-0fcf-4732-9855-00ce27a55d13\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") "
Sep 11 16:39:49.013977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.013959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a985876-0fcf-4732-9855-00ce27a55d13-kserve-provision-location\") pod \"3a985876-0fcf-4732-9855-00ce27a55d13\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") "
Sep 11 16:39:49.014286 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.013985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-cabundle-cert\") pod \"3a985876-0fcf-4732-9855-00ce27a55d13\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") "
Sep 11 16:39:49.014286 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.014070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntl9m\" (UniqueName: \"kubernetes.io/projected/3a985876-0fcf-4732-9855-00ce27a55d13-kube-api-access-ntl9m\") pod \"3a985876-0fcf-4732-9855-00ce27a55d13\" (UID: \"3a985876-0fcf-4732-9855-00ce27a55d13\") "
Sep 11 16:39:49.014393 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.014362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a985876-0fcf-4732-9855-00ce27a55d13" (UID: "3a985876-0fcf-4732-9855-00ce27a55d13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:39:49.014440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.014396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a985876-0fcf-4732-9855-00ce27a55d13" (UID: "3a985876-0fcf-4732-9855-00ce27a55d13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:39:49.014440 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.014367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a985876-0fcf-4732-9855-00ce27a55d13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a985876-0fcf-4732-9855-00ce27a55d13" (UID: "3a985876-0fcf-4732-9855-00ce27a55d13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:49.016218 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.016190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a985876-0fcf-4732-9855-00ce27a55d13" (UID: "3a985876-0fcf-4732-9855-00ce27a55d13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:39:49.016414 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.016390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a985876-0fcf-4732-9855-00ce27a55d13-kube-api-access-ntl9m" (OuterVolumeSpecName: "kube-api-access-ntl9m") pod "3a985876-0fcf-4732-9855-00ce27a55d13" (UID: "3a985876-0fcf-4732-9855-00ce27a55d13"). InnerVolumeSpecName "kube-api-access-ntl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:39:49.115286 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.115243    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntl9m\" (UniqueName: \"kubernetes.io/projected/3a985876-0fcf-4732-9855-00ce27a55d13-kube-api-access-ntl9m\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:39:49.115286 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.115280    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:39:49.115586 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.115295    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a985876-0fcf-4732-9855-00ce27a55d13-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:39:49.115586 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.115310    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a985876-0fcf-4732-9855-00ce27a55d13-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:39:49.115586 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.115323    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a985876-0fcf-4732-9855-00ce27a55d13-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:39:49.550536 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.550493    2578 generic.go:356] "Generic (PLEG): container finished" podID="3a985876-0fcf-4732-9855-00ce27a55d13" containerID="9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3" exitCode=0
Sep 11 16:39:49.550702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.550541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerDied","Data":"9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3"}
Sep 11 16:39:49.550702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.550571    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"
Sep 11 16:39:49.550702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.550637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw" event={"ID":"3a985876-0fcf-4732-9855-00ce27a55d13","Type":"ContainerDied","Data":"56563a3cce524cf987795783c06cae1f55adb42034c86b9ab746c323e9fd097a"}
Sep 11 16:39:49.550702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.550677    2578 scope.go:117] "RemoveContainer" containerID="4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf"
Sep 11 16:39:49.560802 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.560781    2578 scope.go:117] "RemoveContainer" containerID="9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3"
Sep 11 16:39:49.568629 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.568607    2578 scope.go:117] "RemoveContainer" containerID="10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f"
Sep 11 16:39:49.574832 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.574800    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"]
Sep 11 16:39:49.577667 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.577645    2578 scope.go:117] "RemoveContainer" containerID="4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf"
Sep 11 16:39:49.578095 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:39:49.578072    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf\": container with ID starting with 4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf not found: ID does not exist" containerID="4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf"
Sep 11 16:39:49.578187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.578102    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf"} err="failed to get container status \"4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf\": rpc error: code = NotFound desc = could not find container \"4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf\": container with ID starting with 4e691fbf9a12591c051f2334515479e2ab8405d2587e9354e9ea0dc005fc15cf not found: ID does not exist"
Sep 11 16:39:49.578187 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.578124    2578 scope.go:117] "RemoveContainer" containerID="9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3"
Sep 11 16:39:49.578419 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:39:49.578401    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3\": container with ID starting with 9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3 not found: ID does not exist" containerID="9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3"
Sep 11 16:39:49.578460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.578426    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3"} err="failed to get container status \"9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3\": rpc error: code = NotFound desc = could not find container \"9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3\": container with ID starting with 9d3c13a0976324db15de50aa9609013fcf29c25d4dc994e047ab66262b0bb4c3 not found: ID does not exist"
Sep 11 16:39:49.578460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.578441    2578 scope.go:117] "RemoveContainer" containerID="10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f"
Sep 11 16:39:49.578727 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:39:49.578710    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f\": container with ID starting with 10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f not found: ID does not exist" containerID="10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f"
Sep 11 16:39:49.578773 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.578733    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f"} err="failed to get container status \"10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f\": rpc error: code = NotFound desc = could not find container \"10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f\": container with ID starting with 10f9c04748e1c865cf46bf8f53d1e2e5ccfa107f1b071c5928ada91c5464469f not found: ID does not exist"
Sep 11 16:39:49.579032 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.579014    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-pfszw"]
Sep 11 16:39:49.895310 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:49.895278    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" path="/var/lib/kubelet/pods/3a985876-0fcf-4732-9855-00ce27a55d13/volumes"
Sep 11 16:39:51.709725 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.709688    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"]
Sep 11 16:39:51.710210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710135    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="storage-initializer"
Sep 11 16:39:51.710210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710152    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="storage-initializer"
Sep 11 16:39:51.710210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710172    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kube-rbac-proxy"
Sep 11 16:39:51.710210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710180    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kube-rbac-proxy"
Sep 11 16:39:51.710210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710193    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container"
Sep 11 16:39:51.710210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710202    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container"
Sep 11 16:39:51.710533 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710283    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kube-rbac-proxy"
Sep 11 16:39:51.710533 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.710296    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a985876-0fcf-4732-9855-00ce27a55d13" containerName="kserve-container"
Sep 11 16:39:51.715101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.715078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.717507 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.717474    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 11 16:39:51.717646 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.717543    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:39:51.717646 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.717480    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:39:51.717759 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.717746    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:39:51.717759 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.717752    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:39:51.718670 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.718640    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 11 16:39:51.718790 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.718713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:39:51.723903 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.723881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"]
Sep 11 16:39:51.836006 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.835965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.836202 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.836018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.836202 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.836136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.836202 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.836184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.836311 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.836211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5s6f\" (UniqueName: \"kubernetes.io/projected/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-api-access-f5s6f\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.937384 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.937344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.937578 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.937404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.937578 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.937428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5s6f\" (UniqueName: \"kubernetes.io/projected/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-api-access-f5s6f\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.937578 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.937458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.937578 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.937488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.937914 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.937888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.938119 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.938093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.938177 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.938133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.939999 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.939976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:51.946244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:51.946223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5s6f\" (UniqueName: \"kubernetes.io/projected/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-api-access-f5s6f\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-7kt69\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:52.027126 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:52.027003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:52.298749 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:52.298723    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"]
Sep 11 16:39:52.301275 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:39:52.301248    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9b8f10_7b4f_4dba_a42e_d80cda435d85.slice/crio-2da0d99b825806a745f8960fa9de7f0327fff57894f8528fc83218b1051c9aee WatchSource:0}: Error finding container 2da0d99b825806a745f8960fa9de7f0327fff57894f8528fc83218b1051c9aee: Status 404 returned error can't find the container with id 2da0d99b825806a745f8960fa9de7f0327fff57894f8528fc83218b1051c9aee
Sep 11 16:39:52.565824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:52.565726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerStarted","Data":"3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e"}
Sep 11 16:39:52.566007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:52.565836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerStarted","Data":"2da0d99b825806a745f8960fa9de7f0327fff57894f8528fc83218b1051c9aee"}
Sep 11 16:39:56.584982 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:56.584885    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerID="3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e" exitCode=0
Sep 11 16:39:56.584982 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:56.584963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerDied","Data":"3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e"}
Sep 11 16:39:57.590584 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:57.590552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerStarted","Data":"40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99"}
Sep 11 16:39:57.590961 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:57.590613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerStarted","Data":"fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db"}
Sep 11 16:39:57.590961 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:57.590883    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:57.611824 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:57.611775    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podStartSLOduration=2.6859923070000002 podStartE2EDuration="6.611759637s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.92576733s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:39:57.610525278 +0000 UTC m=+1342.228494789" watchObservedRunningTime="2026-09-11 16:39:57.611759637 +0000 UTC m=+1342.229729137"
Sep 11 16:39:58.595368 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:58.595340    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:39:58.596872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:58.596842    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:39:59.599521 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:39:59.599463    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:40:04.604834 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:04.604803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:40:04.605319 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:04.605294    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:40:14.605839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:14.605796    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:40:24.605612 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:24.605565    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:40:34.605774 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:34.605724    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:40:44.605975 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:44.605936    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:40:54.606083 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:40:54.606024    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:41:04.605664 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:04.605615    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:41:14.606803 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:14.606769    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:41:21.847189 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:21.847152    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"]
Sep 11 16:41:21.847668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:21.847436    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" containerID="cri-o://fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db" gracePeriod=30
Sep 11 16:41:21.847668 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:21.847532    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kube-rbac-proxy" containerID="cri-o://40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99" gracePeriod=30
Sep 11 16:41:22.955807 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:22.955775    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerID="40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99" exitCode=2
Sep 11 16:41:22.956212 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:22.955848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerDied","Data":"40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99"}
Sep 11 16:41:24.600547 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:24.600506    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 11 16:41:24.605978 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:24.605947    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 11 16:41:26.895596 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.895567    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:41:26.974377 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.974287    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerID="fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db" exitCode=0
Sep 11 16:41:26.974377 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.974339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"
Sep 11 16:41:26.974377 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.974354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerDied","Data":"fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db"}
Sep 11 16:41:26.974610 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.974413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69" event={"ID":"aa9b8f10-7b4f-4dba-a42e-d80cda435d85","Type":"ContainerDied","Data":"2da0d99b825806a745f8960fa9de7f0327fff57894f8528fc83218b1051c9aee"}
Sep 11 16:41:26.974610 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.974443    2578 scope.go:117] "RemoveContainer" containerID="40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99"
Sep 11 16:41:26.982743 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.982720    2578 scope.go:117] "RemoveContainer" containerID="fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db"
Sep 11 16:41:26.990559 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.990538    2578 scope.go:117] "RemoveContainer" containerID="3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e"
Sep 11 16:41:26.998418 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.998396    2578 scope.go:117] "RemoveContainer" containerID="40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99"
Sep 11 16:41:26.998748 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:41:26.998731    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99\": container with ID starting with 40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99 not found: ID does not exist" containerID="40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99"
Sep 11 16:41:26.998806 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.998759    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99"} err="failed to get container status \"40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99\": rpc error: code = NotFound desc = could not find container \"40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99\": container with ID starting with 40c0b98c75ddd342195d2094f473cad50fed90b1f1a9933dbadae9c5408f5a99 not found: ID does not exist"
Sep 11 16:41:26.998806 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.998777    2578 scope.go:117] "RemoveContainer" containerID="fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db"
Sep 11 16:41:26.999059 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:41:26.999024    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db\": container with ID starting with fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db not found: ID does not exist" containerID="fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db"
Sep 11 16:41:26.999143 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.999069    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db"} err="failed to get container status \"fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db\": rpc error: code = NotFound desc = could not find container \"fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db\": container with ID starting with fb614d0c40e451bde6b4b75646e369818e35cfc3c8a10aecbe7eddc0e8d780db not found: ID does not exist"
Sep 11 16:41:26.999143 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.999095    2578 scope.go:117] "RemoveContainer" containerID="3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e"
Sep 11 16:41:26.999356 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:41:26.999338    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e\": container with ID starting with 3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e not found: ID does not exist" containerID="3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e"
Sep 11 16:41:26.999406 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:26.999363    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e"} err="failed to get container status \"3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e\": rpc error: code = NotFound desc = could not find container \"3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e\": container with ID starting with 3057a311877ec5f97b5272e241c5f68ff1f4d3555ec90d2caebc4918d1e1640e not found: ID does not exist"
Sep 11 16:41:27.063918 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.063882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-rbac-proxy-sar-config\") pod \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") "
Sep 11 16:41:27.064036 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.063923    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kserve-provision-location\") pod \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") "
Sep 11 16:41:27.064036 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.063954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5s6f\" (UniqueName: \"kubernetes.io/projected/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-api-access-f5s6f\") pod \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") "
Sep 11 16:41:27.064036 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.064028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-cabundle-cert\") pod \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") "
Sep 11 16:41:27.064221 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.064104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-proxy-tls\") pod \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\" (UID: \"aa9b8f10-7b4f-4dba-a42e-d80cda435d85\") "
Sep 11 16:41:27.064343 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.064311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa9b8f10-7b4f-4dba-a42e-d80cda435d85" (UID: "aa9b8f10-7b4f-4dba-a42e-d80cda435d85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:41:27.064343 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.064335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa9b8f10-7b4f-4dba-a42e-d80cda435d85" (UID: "aa9b8f10-7b4f-4dba-a42e-d80cda435d85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:41:27.064465 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.064415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa9b8f10-7b4f-4dba-a42e-d80cda435d85" (UID: "aa9b8f10-7b4f-4dba-a42e-d80cda435d85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:41:27.066336 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.066312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa9b8f10-7b4f-4dba-a42e-d80cda435d85" (UID: "aa9b8f10-7b4f-4dba-a42e-d80cda435d85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:41:27.066336 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.066321    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-api-access-f5s6f" (OuterVolumeSpecName: "kube-api-access-f5s6f") pod "aa9b8f10-7b4f-4dba-a42e-d80cda435d85" (UID: "aa9b8f10-7b4f-4dba-a42e-d80cda435d85"). InnerVolumeSpecName "kube-api-access-f5s6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:41:27.165003 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.164967    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5s6f\" (UniqueName: \"kubernetes.io/projected/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-api-access-f5s6f\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:41:27.165003 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.164998    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:41:27.165003 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.165008    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:41:27.165270 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.165019    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:41:27.165270 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.165028    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa9b8f10-7b4f-4dba-a42e-d80cda435d85-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:41:27.296488 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.296425    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"]
Sep 11 16:41:27.298237 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.298203    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-7kt69"]
Sep 11 16:41:27.895108 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:27.895075    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" path="/var/lib/kubelet/pods/aa9b8f10-7b4f-4dba-a42e-d80cda435d85/volumes"
Sep 11 16:41:35.840820 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.840783    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"]
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841102    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kube-rbac-proxy"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841115    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kube-rbac-proxy"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841128    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="storage-initializer"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841134    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="storage-initializer"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841163    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841214    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kube-rbac-proxy"
Sep 11 16:41:35.841294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.841223    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9b8f10-7b4f-4dba-a42e-d80cda435d85" containerName="kserve-container"
Sep 11 16:41:35.844264 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.844234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:35.847090 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.847064    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 11 16:41:35.847318 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.847304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 11 16:41:35.848244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.848224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c2stk\""
Sep 11 16:41:35.848346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.848263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 11 16:41:35.848346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.848271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 11 16:41:35.854436 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.854410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"]
Sep 11 16:41:35.930472 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.930433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:35.930666 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.930488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:35.930666 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:35.930521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vd8x\" (UniqueName: \"kubernetes.io/projected/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-api-access-5vd8x\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.031099 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.031033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.031287 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.031111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vd8x\" (UniqueName: \"kubernetes.io/projected/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-api-access-5vd8x\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.031287 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.031173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.031666 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.031649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.033689 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.033667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.039909 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.039884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vd8x\" (UniqueName: \"kubernetes.io/projected/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-api-access-5vd8x\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-cmxm6\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.156184 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.156135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:36.432529 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:36.432451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"]
Sep 11 16:41:36.437498 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:41:36.437389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d14c2f1_8cd0_4fbf_b56d_5a4c7c5ecb2a.slice/crio-23d327f205f9a29ffc95fdcd8f29ba7356abfd0e5e3bfa0a93c47bbb2473effc WatchSource:0}: Error finding container 23d327f205f9a29ffc95fdcd8f29ba7356abfd0e5e3bfa0a93c47bbb2473effc: Status 404 returned error can't find the container with id 23d327f205f9a29ffc95fdcd8f29ba7356abfd0e5e3bfa0a93c47bbb2473effc
Sep 11 16:41:37.017900 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.017866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" event={"ID":"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a","Type":"ContainerStarted","Data":"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"}
Sep 11 16:41:37.018366 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.017931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" event={"ID":"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a","Type":"ContainerStarted","Data":"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"}
Sep 11 16:41:37.018366 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.017954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" event={"ID":"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a","Type":"ContainerStarted","Data":"23d327f205f9a29ffc95fdcd8f29ba7356abfd0e5e3bfa0a93c47bbb2473effc"}
Sep 11 16:41:37.018366 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.017979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:37.018366 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.018009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:37.019894 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.019874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:37.040098 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:37.038824    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" podStartSLOduration=2.038797917 podStartE2EDuration="2.038797917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:41:37.035251201 +0000 UTC m=+1441.653220725" watchObservedRunningTime="2026-09-11 16:41:37.038797917 +0000 UTC m=+1441.656767420"
Sep 11 16:41:44.031242 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:44.031213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:41:45.900342 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.900311    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"]
Sep 11 16:41:45.903421 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.903401    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:45.905962 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.905936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 11 16:41:45.906127 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.905936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 11 16:41:45.906127 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.905938    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 11 16:41:45.906127 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.905946    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:41:45.913308 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:45.913282    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"]
Sep 11 16:41:46.003427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.003387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.003427 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.003427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.003700 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.003449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdl6v\" (UniqueName: \"kubernetes.io/projected/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-api-access-xdl6v\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.003700 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.003509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.003700 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.003564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.104907 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.104867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.104907 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.104905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.105215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.104925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdl6v\" (UniqueName: \"kubernetes.io/projected/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-api-access-xdl6v\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.105215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.104948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.105215 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.104974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.105215 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:41:46.105114    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 11 16:41:46.105215 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:41:46.105191    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls podName:0ce3e2d5-e8c4-4eb6-a439-d5f065096649 nodeName:}" failed. No retries permitted until 2026-09-11 16:41:46.605170708 +0000 UTC m=+1451.223140191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls") pod "isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" (UID: "0ce3e2d5-e8c4-4eb6-a439-d5f065096649") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 11 16:41:46.105464 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.105362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.105621 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.105602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.105658 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.105624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.114949 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.114918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdl6v\" (UniqueName: \"kubernetes.io/projected/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-api-access-xdl6v\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.608014 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.607974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.610640 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.610618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:46.815582 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:46.815534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:47.094841 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:47.094808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"]
Sep 11 16:41:47.098244 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:41:47.098217    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce3e2d5_e8c4_4eb6_a439_d5f065096649.slice/crio-76dc9f578bedf765ce2a47afda6e492e616ad59ba97f909ff8506a2cf988e12d WatchSource:0}: Error finding container 76dc9f578bedf765ce2a47afda6e492e616ad59ba97f909ff8506a2cf988e12d: Status 404 returned error can't find the container with id 76dc9f578bedf765ce2a47afda6e492e616ad59ba97f909ff8506a2cf988e12d
Sep 11 16:41:48.067038 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:48.066994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerStarted","Data":"e108748cacdc092e7bf616748d0fb483b96fb7345cf20fe3be79db4e081b2481"}
Sep 11 16:41:48.067234 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:48.067126    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerStarted","Data":"76dc9f578bedf765ce2a47afda6e492e616ad59ba97f909ff8506a2cf988e12d"}
Sep 11 16:41:51.081150 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:51.081114    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerID="e108748cacdc092e7bf616748d0fb483b96fb7345cf20fe3be79db4e081b2481" exitCode=0
Sep 11 16:41:51.081513 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:51.081183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerDied","Data":"e108748cacdc092e7bf616748d0fb483b96fb7345cf20fe3be79db4e081b2481"}
Sep 11 16:41:52.089103 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.089035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerStarted","Data":"c12e2c46f271618388287abb9e2a1f20b052a6967252588b8ea48c384f8d7645"}
Sep 11 16:41:52.089560 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.089151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerStarted","Data":"68e68df9d75e208517fb48326b654193efc6f1df1a6cfe4e914647237b70dc31"}
Sep 11 16:41:52.089560 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.089191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerStarted","Data":"d9bded57b7d422cb684261fae47bd7959390c128e3d196773eaafb5c66ff406a"}
Sep 11 16:41:52.089678 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.089631    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:52.089726 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.089693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:52.089726 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.089725    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:52.091027 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.090994    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:41:52.091669 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.091644    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:41:52.111549 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:52.111495    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podStartSLOduration=3.307079622 podStartE2EDuration="7.111479561s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.804399939s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:41:52.109406425 +0000 UTC m=+1456.727375926" watchObservedRunningTime="2026-09-11 16:41:52.111479561 +0000 UTC m=+1456.729449065"
Sep 11 16:41:53.093540 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:53.093488    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:41:53.093994 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:53.093974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:41:58.098475 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:58.098447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:41:58.099184 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:58.099148    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:41:58.099454 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:41:58.099425    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:42:08.099242 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:08.099188    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:42:08.099942 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:08.099697    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:42:18.099926 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:18.099880    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:42:18.100394 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:18.100339    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:42:28.099169 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:28.099119    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:42:28.099664 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:28.099598    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:42:38.099630 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:38.099584    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:42:38.100041 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:38.099997    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:42:48.099039 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:48.098988    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:42:48.099544 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:48.099443    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:42:58.099105 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:58.098992    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:42:58.099606 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:42:58.099389    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:43:08.100467 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:08.100439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:08.102854 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:08.100957    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:20.985225 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:20.985193    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-cmxm6_3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a/kserve-container/0.log"
Sep 11 16:43:21.185413 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.185371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"]
Sep 11 16:43:21.185671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.185613    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" containerID="cri-o://d9bded57b7d422cb684261fae47bd7959390c128e3d196773eaafb5c66ff406a" gracePeriod=30
Sep 11 16:43:21.185671 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.185652    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" containerID="cri-o://c12e2c46f271618388287abb9e2a1f20b052a6967252588b8ea48c384f8d7645" gracePeriod=30
Sep 11 16:43:21.185872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.185705    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" containerID="cri-o://68e68df9d75e208517fb48326b654193efc6f1df1a6cfe4e914647237b70dc31" gracePeriod=30
Sep 11 16:43:21.217474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.217437    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"]
Sep 11 16:43:21.217729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.217702    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kserve-container" containerID="cri-o://794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61" gracePeriod=30
Sep 11 16:43:21.217859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.217725    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kube-rbac-proxy" containerID="cri-o://89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810" gracePeriod=30
Sep 11 16:43:21.461817 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.461793    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:43:21.467817 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467786    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerID="89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810" exitCode=2
Sep 11 16:43:21.467817 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467816    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerID="794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61" exitCode=2
Sep 11 16:43:21.468007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467839    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"
Sep 11 16:43:21.468007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" event={"ID":"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a","Type":"ContainerDied","Data":"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"}
Sep 11 16:43:21.468007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" event={"ID":"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a","Type":"ContainerDied","Data":"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"}
Sep 11 16:43:21.468007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6" event={"ID":"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a","Type":"ContainerDied","Data":"23d327f205f9a29ffc95fdcd8f29ba7356abfd0e5e3bfa0a93c47bbb2473effc"}
Sep 11 16:43:21.468007 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.467985    2578 scope.go:117] "RemoveContainer" containerID="89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"
Sep 11 16:43:21.470307 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.470283    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerID="68e68df9d75e208517fb48326b654193efc6f1df1a6cfe4e914647237b70dc31" exitCode=2
Sep 11 16:43:21.470394 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.470362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerDied","Data":"68e68df9d75e208517fb48326b654193efc6f1df1a6cfe4e914647237b70dc31"}
Sep 11 16:43:21.476023 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.476007    2578 scope.go:117] "RemoveContainer" containerID="794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"
Sep 11 16:43:21.485751 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.485702    2578 scope.go:117] "RemoveContainer" containerID="89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"
Sep 11 16:43:21.486159 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:43:21.486129    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810\": container with ID starting with 89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810 not found: ID does not exist" containerID="89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"
Sep 11 16:43:21.486290 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.486171    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"} err="failed to get container status \"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810\": rpc error: code = NotFound desc = could not find container \"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810\": container with ID starting with 89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810 not found: ID does not exist"
Sep 11 16:43:21.486290 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.486195    2578 scope.go:117] "RemoveContainer" containerID="794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"
Sep 11 16:43:21.486540 ip-10-0-135-5 kubenswrapper[2578]: E0911 16:43:21.486513    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61\": container with ID starting with 794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61 not found: ID does not exist" containerID="794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"
Sep 11 16:43:21.486660 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.486550    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"} err="failed to get container status \"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61\": rpc error: code = NotFound desc = could not find container \"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61\": container with ID starting with 794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61 not found: ID does not exist"
Sep 11 16:43:21.486660 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.486573    2578 scope.go:117] "RemoveContainer" containerID="89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"
Sep 11 16:43:21.486872 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.486848    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810"} err="failed to get container status \"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810\": rpc error: code = NotFound desc = could not find container \"89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810\": container with ID starting with 89f20ac33476095b65331cc9ef8fac9aaf7fb2f0445cac57380f8c3b59b90810 not found: ID does not exist"
Sep 11 16:43:21.486931 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.486876    2578 scope.go:117] "RemoveContainer" containerID="794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"
Sep 11 16:43:21.487210 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.487193    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61"} err="failed to get container status \"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61\": rpc error: code = NotFound desc = could not find container \"794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61\": container with ID starting with 794f93bc9c3340f4fc5d2d562224ee625bd0a0a666e77a9c71906a6fa8486a61 not found: ID does not exist"
Sep 11 16:43:21.555460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.555423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vd8x\" (UniqueName: \"kubernetes.io/projected/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-api-access-5vd8x\") pod \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") "
Sep 11 16:43:21.555460 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.555459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-rbac-proxy-sar-config\") pod \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") "
Sep 11 16:43:21.555738 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.555581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-proxy-tls\") pod \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\" (UID: \"3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a\") "
Sep 11 16:43:21.555895 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.555870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" (UID: "3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:43:21.557795 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.557766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-api-access-5vd8x" (OuterVolumeSpecName: "kube-api-access-5vd8x") pod "3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" (UID: "3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a"). InnerVolumeSpecName "kube-api-access-5vd8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:21.557904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.557793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" (UID: "3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:43:21.656977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.656937    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:21.656977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.656969    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vd8x\" (UniqueName: \"kubernetes.io/projected/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-api-access-5vd8x\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:21.656977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.656978    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:21.789176 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.789141    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"]
Sep 11 16:43:21.795018 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.794991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-cmxm6"]
Sep 11 16:43:21.894911 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:21.894883    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" path="/var/lib/kubelet/pods/3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a/volumes"
Sep 11 16:43:23.094615 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:23.094571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 11 16:43:26.498306 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:26.498273    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerID="d9bded57b7d422cb684261fae47bd7959390c128e3d196773eaafb5c66ff406a" exitCode=0
Sep 11 16:43:26.498690 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:26.498348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerDied","Data":"d9bded57b7d422cb684261fae47bd7959390c128e3d196773eaafb5c66ff406a"}
Sep 11 16:43:28.093809 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:28.093767    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 11 16:43:28.099547 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:28.099495    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:43:28.101250 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:28.101228    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:43:33.093908 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:33.093859    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 11 16:43:33.094396 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:33.093996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:38.094176 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.094127    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 11 16:43:38.099942 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.099910    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:43:38.101233 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.101211    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:43:38.399352 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.399320    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:38.428376 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.428339    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:38.454432 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.454409    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:38.505095 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:38.505065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:39.331652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:39.331624    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:39.343985 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:39.343959    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:39.354008 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:39.353984    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:39.364604 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:39.364569    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:40.211729 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:40.211696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:40.238197 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:40.238168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:40.250623 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:40.250601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:40.261439 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:40.261414    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:41.096323 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.096288    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:41.108977 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.108936    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:41.119474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.119444    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:41.130332 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.130291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:41.945465 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.945439    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:41.958919 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.958883    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:41.969237 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.969191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:41.980299 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:41.980272    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:42.793640 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:42.793592    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:42.806039 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:42.806006    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:42.815975 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:42.815953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:42.825647 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:42.825616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:43.094182 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:43.094093    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 11 16:43:43.660213 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:43.660181    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:43.671679 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:43.671641    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:43.681409 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:43.681379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:43.691522 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:43.691501    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:44.524236 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:44.524203    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:44.536795 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:44.536768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:44.547943 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:44.547909    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:44.559087 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:44.559040    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:45.414823 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:45.414794    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:45.446511 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:45.446477    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:45.476428 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:45.476399    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:45.487469 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:45.487443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:46.305904 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:46.305873    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:46.318196 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:46.318158    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:46.328269 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:46.328245    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:46.338438 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:46.338406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:47.181984 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:47.181958    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:47.194437 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:47.194413    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:47.205335 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:47.205311    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:47.216784 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:47.216757    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:48.047483 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.047450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:48.059268 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.059227    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:48.068803 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.068779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:48.078774 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.078750    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:48.094204 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.094170    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 11 16:43:48.100028 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.100000    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:43:48.100185 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.100128    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:48.101261 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.101234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 11 16:43:48.101366 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.101317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:48.920649 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.920602    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:48.932306 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.932282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:48.942829 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.942801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:48.953054 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:48.953022    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:49.829565 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:49.829533    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kserve-container/0.log"
Sep 11 16:43:49.840773 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:49.840739    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/kube-rbac-proxy/0.log"
Sep 11 16:43:49.854550 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:49.854515    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/agent/0.log"
Sep 11 16:43:49.866039 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:49.866004    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf_0ce3e2d5-e8c4-4eb6-a439-d5f065096649/storage-initializer/0.log"
Sep 11 16:43:51.611472 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:51.611444    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerID="c12e2c46f271618388287abb9e2a1f20b052a6967252588b8ea48c384f8d7645" exitCode=0
Sep 11 16:43:51.611841 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:51.611502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerDied","Data":"c12e2c46f271618388287abb9e2a1f20b052a6967252588b8ea48c384f8d7645"}
Sep 11 16:43:51.837002 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:51.836978    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:52.002634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-rbac-proxy-sar-config\") pod \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") "
Sep 11 16:43:52.002634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002587    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdl6v\" (UniqueName: \"kubernetes.io/projected/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-api-access-xdl6v\") pod \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") "
Sep 11 16:43:52.002634 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls\") pod \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") "
Sep 11 16:43:52.002937 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kserve-provision-location\") pod \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") "
Sep 11 16:43:52.002937 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-cabundle-cert\") pod \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\" (UID: \"0ce3e2d5-e8c4-4eb6-a439-d5f065096649\") "
Sep 11 16:43:52.003037 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0ce3e2d5-e8c4-4eb6-a439-d5f065096649" (UID: "0ce3e2d5-e8c4-4eb6-a439-d5f065096649"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:43:52.003037 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.002994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ce3e2d5-e8c4-4eb6-a439-d5f065096649" (UID: "0ce3e2d5-e8c4-4eb6-a439-d5f065096649"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:52.003145 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.003097    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0ce3e2d5-e8c4-4eb6-a439-d5f065096649" (UID: "0ce3e2d5-e8c4-4eb6-a439-d5f065096649"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:43:52.004879 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.004858    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ce3e2d5-e8c4-4eb6-a439-d5f065096649" (UID: "0ce3e2d5-e8c4-4eb6-a439-d5f065096649"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:43:52.004943 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.004895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-api-access-xdl6v" (OuterVolumeSpecName: "kube-api-access-xdl6v") pod "0ce3e2d5-e8c4-4eb6-a439-d5f065096649" (UID: "0ce3e2d5-e8c4-4eb6-a439-d5f065096649"). InnerVolumeSpecName "kube-api-access-xdl6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:52.104070 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.104002    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kserve-provision-location\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:52.104257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.104086    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-cabundle-cert\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:52.104257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.104102    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:52.104257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.104113    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdl6v\" (UniqueName: \"kubernetes.io/projected/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-kube-api-access-xdl6v\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:52.104257 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.104125    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ce3e2d5-e8c4-4eb6-a439-d5f065096649-proxy-tls\") on node \"ip-10-0-135-5.ec2.internal\" DevicePath \"\""
Sep 11 16:43:52.618515 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.618478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf" event={"ID":"0ce3e2d5-e8c4-4eb6-a439-d5f065096649","Type":"ContainerDied","Data":"76dc9f578bedf765ce2a47afda6e492e616ad59ba97f909ff8506a2cf988e12d"}
Sep 11 16:43:52.618991 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.618554    2578 scope.go:117] "RemoveContainer" containerID="c12e2c46f271618388287abb9e2a1f20b052a6967252588b8ea48c384f8d7645"
Sep 11 16:43:52.618991 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.618492    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"
Sep 11 16:43:52.627244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.627227    2578 scope.go:117] "RemoveContainer" containerID="68e68df9d75e208517fb48326b654193efc6f1df1a6cfe4e914647237b70dc31"
Sep 11 16:43:52.634775 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.634754    2578 scope.go:117] "RemoveContainer" containerID="d9bded57b7d422cb684261fae47bd7959390c128e3d196773eaafb5c66ff406a"
Sep 11 16:43:52.642464 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.642436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"]
Sep 11 16:43:52.643064 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.643024    2578 scope.go:117] "RemoveContainer" containerID="e108748cacdc092e7bf616748d0fb483b96fb7345cf20fe3be79db4e081b2481"
Sep 11 16:43:52.646131 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:52.646107    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-778ff5f68f-jhcpf"]
Sep 11 16:43:53.895080 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.895025    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" path="/var/lib/kubelet/pods/0ce3e2d5-e8c4-4eb6-a439-d5f065096649/volumes"
Sep 11 16:43:53.931332 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931298    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9j26t/must-gather-52dwf"]
Sep 11 16:43:53.931604 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931591    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kserve-container"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931606    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kserve-container"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931613    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kube-rbac-proxy"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931618    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kube-rbac-proxy"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931629    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931634    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931645    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="storage-initializer"
Sep 11 16:43:53.931651 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931650    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="storage-initializer"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931655    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931660    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931669    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931674    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931716    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kserve-container"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931725    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="agent"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931730    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d14c2f1-8cd0-4fbf-b56d-5a4c7c5ecb2a" containerName="kube-rbac-proxy"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931735    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kube-rbac-proxy"
Sep 11 16:43:53.931859 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.931741    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce3e2d5-e8c4-4eb6-a439-d5f065096649" containerName="kserve-container"
Sep 11 16:43:53.936532 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.936513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:53.939113 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.939083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9j26t\"/\"openshift-service-ca.crt\""
Sep 11 16:43:53.939246 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.939182    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9j26t\"/\"kube-root-ca.crt\""
Sep 11 16:43:53.940389 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.940296    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9j26t\"/\"default-dockercfg-9ckph\""
Sep 11 16:43:53.942421 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:53.942401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9j26t/must-gather-52dwf"]
Sep 11 16:43:54.120781 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.120742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2g8\" (UniqueName: \"kubernetes.io/projected/13b79679-41a8-4d59-a5d8-167e728f2b48-kube-api-access-5m2g8\") pod \"must-gather-52dwf\" (UID: \"13b79679-41a8-4d59-a5d8-167e728f2b48\") " pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.120979 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.120818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13b79679-41a8-4d59-a5d8-167e728f2b48-must-gather-output\") pod \"must-gather-52dwf\" (UID: \"13b79679-41a8-4d59-a5d8-167e728f2b48\") " pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.221364 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.221275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13b79679-41a8-4d59-a5d8-167e728f2b48-must-gather-output\") pod \"must-gather-52dwf\" (UID: \"13b79679-41a8-4d59-a5d8-167e728f2b48\") " pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.221364 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.221336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2g8\" (UniqueName: \"kubernetes.io/projected/13b79679-41a8-4d59-a5d8-167e728f2b48-kube-api-access-5m2g8\") pod \"must-gather-52dwf\" (UID: \"13b79679-41a8-4d59-a5d8-167e728f2b48\") " pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.221581 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.221562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13b79679-41a8-4d59-a5d8-167e728f2b48-must-gather-output\") pod \"must-gather-52dwf\" (UID: \"13b79679-41a8-4d59-a5d8-167e728f2b48\") " pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.230192 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.230165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2g8\" (UniqueName: \"kubernetes.io/projected/13b79679-41a8-4d59-a5d8-167e728f2b48-kube-api-access-5m2g8\") pod \"must-gather-52dwf\" (UID: \"13b79679-41a8-4d59-a5d8-167e728f2b48\") " pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.257299 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.257260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9j26t/must-gather-52dwf"
Sep 11 16:43:54.528344 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.528196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9j26t/must-gather-52dwf"]
Sep 11 16:43:54.530717 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:43:54.530692    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13b79679_41a8_4d59_a5d8_167e728f2b48.slice/crio-2f9524060464103bc6a7d24866f4e94a0857928fd110bf1265be6995d2790318 WatchSource:0}: Error finding container 2f9524060464103bc6a7d24866f4e94a0857928fd110bf1265be6995d2790318: Status 404 returned error can't find the container with id 2f9524060464103bc6a7d24866f4e94a0857928fd110bf1265be6995d2790318
Sep 11 16:43:54.532898 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.532881    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:43:54.628932 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:54.628896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/must-gather-52dwf" event={"ID":"13b79679-41a8-4d59-a5d8-167e728f2b48","Type":"ContainerStarted","Data":"2f9524060464103bc6a7d24866f4e94a0857928fd110bf1265be6995d2790318"}
Sep 11 16:43:55.635535 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:55.635442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/must-gather-52dwf" event={"ID":"13b79679-41a8-4d59-a5d8-167e728f2b48","Type":"ContainerStarted","Data":"024ed60385e5056951ad14c13f995feccc1f6ca672cd0280acede0c056a93125"}
Sep 11 16:43:55.635535 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:55.635504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/must-gather-52dwf" event={"ID":"13b79679-41a8-4d59-a5d8-167e728f2b48","Type":"ContainerStarted","Data":"468581489e1cd6e4c26e3f03eeeeaa2238bb12a88431f31c0dd8231dffacf4c0"}
Sep 11 16:43:55.653726 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:55.653666    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9j26t/must-gather-52dwf" podStartSLOduration=1.80506237 podStartE2EDuration="2.653644454s" totalImagesPullingTime="848.582084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:43:55.651294608 +0000 UTC m=+1580.269264111" watchObservedRunningTime="2026-09-11 16:43:55.653644454 +0000 UTC m=+1580.271613955"
Sep 11 16:43:56.900502 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:56.900470    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p5q6j_179f2ddc-df25-46bf-b640-94826d4f434d/global-pull-secret-syncer/0.log"
Sep 11 16:43:57.055090 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:57.055059    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k6gfq_3da31bc3-f199-4904-9c3d-de3d60ff4815/konnectivity-agent/0.log"
Sep 11 16:43:57.074839 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:43:57.074814    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-5.ec2.internal_243f33b0da41e521566de787222b729f/haproxy/0.log"
Sep 11 16:44:01.156892 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.156851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/alertmanager/0.log"
Sep 11 16:44:01.188155 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.188126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/config-reloader/0.log"
Sep 11 16:44:01.216404 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.216371    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/kube-rbac-proxy-web/0.log"
Sep 11 16:44:01.244521 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.244495    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/kube-rbac-proxy/0.log"
Sep 11 16:44:01.270432 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.270406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/kube-rbac-proxy-metric/0.log"
Sep 11 16:44:01.299652 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.299630    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/prom-label-proxy/0.log"
Sep 11 16:44:01.323983 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.323945    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dcb1c722-1f26-40d4-b9eb-75919fec6ccb/init-config-reloader/0.log"
Sep 11 16:44:01.547294 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.547215    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlnft_ef54c37d-b09a-4c08-b617-a286b8131213/node-exporter/0.log"
Sep 11 16:44:01.568967 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.568916    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlnft_ef54c37d-b09a-4c08-b617-a286b8131213/kube-rbac-proxy/0.log"
Sep 11 16:44:01.591879 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:01.591849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hlnft_ef54c37d-b09a-4c08-b617-a286b8131213/init-textfile/0.log"
Sep 11 16:44:05.397723 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.397690    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"]
Sep 11 16:44:05.403244 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.403218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.411115 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.411089    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"]
Sep 11 16:44:05.518649 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.518611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-lib-modules\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.518946 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.518918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-proc\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.519113 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.518960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phb8k\" (UniqueName: \"kubernetes.io/projected/ae3f89cf-bbd6-40ff-95bd-194c13af4383-kube-api-access-phb8k\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.519113 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.519001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-sys\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.519113 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.519024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-podres\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.619717 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.619677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phb8k\" (UniqueName: \"kubernetes.io/projected/ae3f89cf-bbd6-40ff-95bd-194c13af4383-kube-api-access-phb8k\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.620006 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.619987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-sys\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.620296 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-podres\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.620459 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-podres\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.620518 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-sys\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.620665 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-lib-modules\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.620897 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-proc\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.621015 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-proc\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.621209 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.620840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae3f89cf-bbd6-40ff-95bd-194c13af4383-lib-modules\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.628278 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.628250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phb8k\" (UniqueName: \"kubernetes.io/projected/ae3f89cf-bbd6-40ff-95bd-194c13af4383-kube-api-access-phb8k\") pod \"perf-node-gather-daemonset-4jdnv\" (UID: \"ae3f89cf-bbd6-40ff-95bd-194c13af4383\") " pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:05.717766 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:05.717682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:06.015963 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.015934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"]
Sep 11 16:44:06.020248 ip-10-0-135-5 kubenswrapper[2578]: W0911 16:44:06.019621    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podae3f89cf_bbd6_40ff_95bd_194c13af4383.slice/crio-c63db4ba49bb17fb701dcd6b5fbd77270c013efe3c24fa4ae96aa8af1005caf0 WatchSource:0}: Error finding container c63db4ba49bb17fb701dcd6b5fbd77270c013efe3c24fa4ae96aa8af1005caf0: Status 404 returned error can't find the container with id c63db4ba49bb17fb701dcd6b5fbd77270c013efe3c24fa4ae96aa8af1005caf0
Sep 11 16:44:06.099474 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.099443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g55bs_febf625b-e5c1-417f-98cf-3c3ef5988d5a/dns/0.log"
Sep 11 16:44:06.123515 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.123468    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g55bs_febf625b-e5c1-417f-98cf-3c3ef5988d5a/kube-rbac-proxy/0.log"
Sep 11 16:44:06.248793 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.248763    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b779c_a0f6a39a-b4d2-420b-86d6-6535644a7dce/dns-node-resolver/0.log"
Sep 11 16:44:06.707064 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.706143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv" event={"ID":"ae3f89cf-bbd6-40ff-95bd-194c13af4383","Type":"ContainerStarted","Data":"7587a4b08ffdb0282d5243922f29404cd011e40c1bd834242d70e0dd209971ef"}
Sep 11 16:44:06.707064 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.706249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv" event={"ID":"ae3f89cf-bbd6-40ff-95bd-194c13af4383","Type":"ContainerStarted","Data":"c63db4ba49bb17fb701dcd6b5fbd77270c013efe3c24fa4ae96aa8af1005caf0"}
Sep 11 16:44:06.707064 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.706995    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:06.725346 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.725281    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv" podStartSLOduration=1.7252571030000001 podStartE2EDuration="1.725257103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:06.723522934 +0000 UTC m=+1591.341492437" watchObservedRunningTime="2026-09-11 16:44:06.725257103 +0000 UTC m=+1591.343226604"
Sep 11 16:44:06.829858 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:06.829821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c5vjm_fc1cdc55-6865-430d-8810-8d306526177f/node-ca/0.log"
Sep 11 16:44:08.015910 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:08.015875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vsb48_fc7eab1c-9b8f-4a35-86b8-832abe802552/serve-healthcheck-canary/0.log"
Sep 11 16:44:08.466207 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:08.466179    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5pxx_3b5847d0-27fe-49e6-bb07-127e2fd8b4b6/kube-rbac-proxy/0.log"
Sep 11 16:44:08.484884 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:08.484843    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5pxx_3b5847d0-27fe-49e6-bb07-127e2fd8b4b6/exporter/0.log"
Sep 11 16:44:08.505059 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:08.505016    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5pxx_3b5847d0-27fe-49e6-bb07-127e2fd8b4b6/extractor/0.log"
Sep 11 16:44:10.917192 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:10.917158    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-656b75fc8d-b6qrd_57d7bbac-98e2-447d-aa8e-9097bdaee58d/manager/0.log"
Sep 11 16:44:10.954067 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:10.954024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-lbx8v_7a3e6100-5b0b-4685-b5c3-ae9ebb82e1c5/manager/0.log"
Sep 11 16:44:10.971784 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:10.971756    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-7l97w_fe97ef8c-b207-4410-ad9d-a3e1a9cc1879/server/0.log"
Sep 11 16:44:11.199189 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:11.199113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-tkhrk_c6033440-bb82-4e8a-bf63-e7e2d61d09d3/manager/0.log"
Sep 11 16:44:11.287130 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:11.287096    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-rfgbr_44d6fc10-93fc-4a7a-abcc-d620ea89c42f/seaweedfs/0.log"
Sep 11 16:44:13.728568 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:13.728540    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9j26t/perf-node-gather-daemonset-4jdnv"
Sep 11 16:44:16.798042 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.797961    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/kube-multus-additional-cni-plugins/0.log"
Sep 11 16:44:16.817570 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.817535    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/egress-router-binary-copy/0.log"
Sep 11 16:44:16.836907 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.836877    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/cni-plugins/0.log"
Sep 11 16:44:16.856742 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.856704    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/bond-cni-plugin/0.log"
Sep 11 16:44:16.875925 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.875896    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/routeoverride-cni/0.log"
Sep 11 16:44:16.894673 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.894644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/whereabouts-cni-bincopy/0.log"
Sep 11 16:44:16.912900 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:16.912874    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-86z74_a0aef5d9-403b-4e8e-b0c0-5b27b5b89bd0/whereabouts-cni/0.log"
Sep 11 16:44:17.280402 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:17.280262    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mvn6l_815ae807-3454-424e-8beb-77f2fdce09bf/kube-multus/0.log"
Sep 11 16:44:17.436383 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:17.436355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s2ks4_51b154f6-27bb-4554-b269-0cfa1de4bda7/network-metrics-daemon/0.log"
Sep 11 16:44:17.454409 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:17.454383    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s2ks4_51b154f6-27bb-4554-b269-0cfa1de4bda7/kube-rbac-proxy/0.log"
Sep 11 16:44:18.415101 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.415015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/ovn-controller/0.log"
Sep 11 16:44:18.430577 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.430542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/ovn-acl-logging/0.log"
Sep 11 16:44:18.453182 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.453151    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/kube-rbac-proxy-node/0.log"
Sep 11 16:44:18.473702 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.473668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 16:44:18.492820 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.492792    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/northd/0.log"
Sep 11 16:44:18.511459 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.511430    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/nbdb/0.log"
Sep 11 16:44:18.531607 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.531571    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/sbdb/0.log"
Sep 11 16:44:18.585784 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:18.585734    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqb2x_02b9217e-d171-4f08-95fa-90bd09c6675a/ovnkube-controller/0.log"
Sep 11 16:44:19.843420 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:19.843387    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-8gctq_e2f260be-4aab-4a68-87d2-35cf264b3d1d/network-check-target-container/0.log"
Sep 11 16:44:20.936866 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:20.936839    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bg5xb_39aa53cb-d4d0-422a-9f4e-abc9b2d8e4a6/iptables-alerter/0.log"
Sep 11 16:44:21.743865 ip-10-0-135-5 kubenswrapper[2578]: I0911 16:44:21.743832    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-p7mzb_f0e6493f-bb8e-4c87-8f2e-9bb5a6861912/tuned/0.log"