Sep 18 16:16:26.455956 ip-10-0-136-96 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:16:29.998789 ip-10-0-136-96 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:29.998789 ip-10-0-136-96 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:16:29.998789 ip-10-0-136-96 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:29.998789 ip-10-0-136-96 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:16:29.998789 ip-10-0-136-96 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:29.999495 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:29.998864    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:16:30.005149 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005120    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:30.005149 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005143    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:30.005149 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005147    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:30.005149 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005150    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:30.005149 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005154    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:30.005149 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005158    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005163    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005167    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005170    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005173    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005175    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005179    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005182    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005185    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005188    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005190    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005193    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005196    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005199    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005201    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005204    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005206    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005209    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005212    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005215    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:30.005404 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005219    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005221    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005227    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005230    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005232    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005235    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005238    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005242    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005244    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005247    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005249    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005252    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005256    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005261    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005268    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005272    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005277    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005281    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005284    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005286    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:30.005922 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005289    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005292    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005295    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005297    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005300    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005302    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005306    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005308    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005311    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005313    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005316    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005319    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005321    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005324    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005326    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005330    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005333    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005335    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005338    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005340    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:30.006445 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005342    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005345    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005347    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005350    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005352    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005355    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005358    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005360    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005363    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005366    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005368    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005370    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005373    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005375    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005378    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005380    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005383    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005386    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005389    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:30.006945 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005392    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005394    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005397    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005399    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005402    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005405    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005408    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005410    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005413    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005416    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005419    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005421    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005424    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005426    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.005429    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006179    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006186    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006190    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006192    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:30.007440 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006197    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006200    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006204    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006207    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006210    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006213    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006216    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006219    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006221    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006224    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006226    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006229    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006232    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006240    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006243    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006245    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006248    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006251    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006253    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006256    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:30.007927 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006260    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006263    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006266    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006268    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006271    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006273    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006276    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006279    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006281    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006284    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006286    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006289    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006292    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006295    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006298    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006300    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006303    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006306    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006310    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:30.008423 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006313    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006317    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006320    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006323    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006326    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006329    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006332    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006340    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006343    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006346    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006349    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006352    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006355    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006358    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006361    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006364    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006366    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006370    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006372    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:30.008907 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006375    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006377    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006380    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006383    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006385    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006388    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006391    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006394    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006397    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006400    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006402    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006405    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006407    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006410    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006413    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006415    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006418    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006421    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006425    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:30.009356 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006428    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006430    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006434    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006437    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006441    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006444    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006447    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006449    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006452    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006455    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006457    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006460    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006463    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006466    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006469    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006472    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006474    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006477    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006892    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006897    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:30.009832 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006900    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006905    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006909    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006912    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006916    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006919    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006922    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006925    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006928    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006931    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006934    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006937    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006940    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006943    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006946    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006949    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006952    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006955    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006958    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:30.010316 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006961    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006964    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006966    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006969    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006972    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006975    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006977    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006980    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006983    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006985    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006988    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006990    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006994    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006997    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.006999    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007002    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007004    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007007    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007010    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007012    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:30.010822 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007015    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007018    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007020    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007023    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007025    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007028    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007031    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007033    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007039    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007041    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007044    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007046    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007049    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007052    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007054    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007057    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007060    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007063    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007065    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007068    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:30.011293 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007071    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007073    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007076    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007079    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007081    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007084    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007087    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007090    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007092    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007095    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007097    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007100    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007103    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007105    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007108    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007110    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007113    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007116    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007118    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007121    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:30.011830 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007123    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007126    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007129    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007133    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007137    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007140    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007142    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007145    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007148    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007151    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007154    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007156    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007159    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007162    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007164    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007167    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007169    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007172    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007250    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007287    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:16:30.012311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007294    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007299    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007303    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007306    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007311    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007316    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007319    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007322    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007326    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007329    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007332    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007335    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007338    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007341    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007344    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007347    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007352    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007355    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007358    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007361    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007370    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007373    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007376    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:16:30.012820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007379    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007382    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007385    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007389    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007392    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007395    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007397    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007402    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007405    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007408    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007411    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007414    2563 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007417    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007421    2563 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007424    2563 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007427    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007430    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007433    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007436    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007439    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007442    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007445    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007448    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007451    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007454    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:16:30.013404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007457    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007460    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007463    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007466    2563 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007471    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007477    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007480    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007483    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007486    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007489    2563 flags.go:64] FLAG: --help="false"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007491    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007495    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007498    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007501    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007504    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007507    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007510    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007513    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007525    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007529    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007532    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007535    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007538    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007541    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:16:30.014043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007544    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007547    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007550    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007553    2563 flags.go:64] FLAG: --lock-file=""
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007571    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007574    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007577    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007583    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007586    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007589    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007592    2563 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007595    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007600    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007604    2563 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007607    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007611    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007614    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007619    2563 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007622    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007625    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007628    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007630    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007633    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007636    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007639    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:16:30.014662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007647    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007650    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007653    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007656    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007659    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007664    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007667    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007670    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007673    2563 flags.go:64] FLAG: --port="10250"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007676    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007679    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ec324f33533c9bd"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007682    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007685    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007688    2563 flags.go:64] FLAG: --register-node="true"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007690    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007694    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007698    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007700    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007703    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007708    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007711    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007715    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007718    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007721    2563 flags.go:64] FLAG: --runonce="false"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007724    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:16:30.015279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007727    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007730    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007733    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007735    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007738    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007741    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007744    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007747    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007750    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007753    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007755    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007758    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007761    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007764    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007769    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007772    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007774    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007778    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007781    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007784    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007786    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007789    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007792    2563 flags.go:64] FLAG: --v="2"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007796    2563 flags.go:64] FLAG: --version="false"
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007801    2563 flags.go:64] FLAG: --vmodule=""
Sep 18 16:16:30.015929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007804    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.007809    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007926    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007932    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007935    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007938    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007941    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007946    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007950    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007963    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007966    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007968    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007971    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007974    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007976    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007979    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007982    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007986    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007990    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:30.016529 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007993    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007996    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.007998    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008001    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008003    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008006    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008008    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008011    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008013    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008016    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008019    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008021    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008024    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008027    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008031    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008033    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008037    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008039    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008042    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008044    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:30.017012 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008047    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008049    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008052    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008054    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008057    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008059    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008061    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008064    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008067    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008069    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008072    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008074    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008077    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008079    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008081    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008084    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008086    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008089    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008091    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:30.017496 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008095    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008099    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008102    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008104    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008108    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008111    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008113    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008118    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008120    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008124    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008126    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008129    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008131    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008134    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008137    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008139    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008142    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008145    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008147    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008150    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:30.017992 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008152    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008155    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008157    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008160    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008162    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008165    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008167    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008170    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008172    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008175    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008178    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008180    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008183    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008185    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008188    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008190    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008192    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008195    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008197    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008201    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:30.018476 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008204    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008207    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.008210    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.009048    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016093    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016109    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016133    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016142    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016196    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016200    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016204    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016207    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016210    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016213    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:30.018973 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016216    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016219    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016222    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016224    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016227    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016230    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016233    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016236    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016238    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016241    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016243    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016246    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016248    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016251    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016254    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016256    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016259    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016262    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016264    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016266    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:30.019345 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016269    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016272    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016274    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016279    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016282    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016284    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016287    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016289    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016292    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016294    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016297    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016299    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016302    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016304    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016308    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016311    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016314    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016317    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016320    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016323    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:30.019943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016325    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016328    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016330    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016333    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016335    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016338    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016340    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016343    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016345    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016348    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016350    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016352    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016355    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016357    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016360    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016364    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016367    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016370    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016373    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:30.020437 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016377    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016380    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016383    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016386    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016389    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016392    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016394    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016397    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016399    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016402    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016406    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016409    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016412    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016414    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016417    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016420    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016422    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016425    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016428    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:30.020943 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016430    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016433    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016436    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016438    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016440    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016443    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016445    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016448    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016451    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016454    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016456    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016460    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016462    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016465    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016467    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:30.021397 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016472    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016601    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016608    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016611    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016614    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016616    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016619    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016622    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016624    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016627    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016630    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016632    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016635    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016638    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016640    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016643    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016645    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016648    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016650    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016653    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:30.021776 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016655    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016658    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016660    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016663    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016666    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016668    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016671    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016674    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016676    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016680    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016683    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016686    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016688    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016691    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016693    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016696    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016699    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016701    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016704    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016706    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:30.022245 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016709    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016712    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016716    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016719    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016722    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016725    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016727    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016730    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016733    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016736    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016738    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016741    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016743    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016746    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016748    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016752    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016754    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016757    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016760    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:30.022731 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016763    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016765    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016768    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016771    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016773    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016776    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016779    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016781    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016784    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016786    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016789    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016791    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016794    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016798    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016801    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016804    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016806    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016809    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016811    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:30.023182 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016814    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016816    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016819    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016822    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016824    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016827    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016829    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016832    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016835    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016837    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016840    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016842    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016845    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016848    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016851    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016853    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016857    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016860    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016863    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016866    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:30.023737 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016868    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:30.016871    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016875    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.016992    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.018964    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.020522    2563 server.go:1019] "Starting client certificate rotation"
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.020628    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.021141    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:30.024222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.023840    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:30.024997 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.024979    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:30.029595 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.029577    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:16:30.031016 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.031003    2563 log.go:25] "Validated CRI v1 image API"
Sep 18 16:16:30.031766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.031752    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:16:30.034096 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.034072    2563 fs.go:135] Filesystem UUIDs: map[6141db11-829d-4e29-8098-08ddf85cb9da:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7bf126ae-bf17-4990-9f78-0ca6166201a6:/dev/nvme0n1p4]
Sep 18 16:16:30.034163 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.034095    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:16:30.039420 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.039311    2563 manager.go:216] Machine: {Timestamp:2026-09-18 16:16:30.038849056 +0000 UTC m=+2.039109870 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106938 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec200446db61d8fd6283b658fd77ce52 SystemUUID:ec200446-db61-d8fd-6283-b658fd77ce52 BootID:43f03c25-16af-488e-a47d-6cacfd3cd7d6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:96:6a:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:96:6a:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:58:11:52:5a:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:16:30.039420 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.039408    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:16:30.039573 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.039487    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:16:30.041004 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.040978    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:16:30.041134 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.041007    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:16:30.042140 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.042129    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:16:30.042176 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.042143    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:16:30.042176 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.042150    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:30.042176 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.042161    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:30.042252 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.042229    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:16:30.045697 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045685    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:30.045814 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045804    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:16:30.045851 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045838    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:16:30.045881 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045852    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:16:30.045881 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045865    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:16:30.045881 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045873    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:16:30.045881 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.045881    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:16:30.046209 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.046197    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:30.046263 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.046221    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:30.048980 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.048966    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:16:30.051205 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.051192    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:16:30.051246 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.051227    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:16:30.052447 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052434    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:16:30.052508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052452    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:16:30.052508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052459    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:16:30.052508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052464    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:16:30.052508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052489    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:16:30.052508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052496    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:16:30.052508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052510    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052515    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052522    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052527    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052539    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052545    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052592    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:16:30.052696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.052598    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:16:30.056418 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.056405    2563 server.go:1268] "Started kubelet"
Sep 18 16:16:30.056544 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.056490    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:16:30.056615 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.056520    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:16:30.056615 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.056611    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:16:30.057340 ip-10-0-136-96 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:16:30.057926 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.057742    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:16:30.058358 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.058340    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:16:30.060047 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.060022    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:16:30.060118 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.060061    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:16:30.060118 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.060078    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:16:30.061920 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.061885    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:16:30.063879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.063861    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:16:30.063879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.063874    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:30.064023 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.063928    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.064336 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.064320    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:16:30.064416 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.064346    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:16:30.064416 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.064320    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:16:30.066189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066035    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:16:30.066189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066052    2563 factory.go:55] Registering systemd factory
Sep 18 16:16:30.066189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066062    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:16:30.066379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066232    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:16:30.066379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066245    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:16:30.066379 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.066362    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:16:30.066510 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066383    2563 factory.go:153] Registering CRI-O factory
Sep 18 16:16:30.066510 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066397    2563 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:16:30.066510 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066418    2563 factory.go:103] Registering Raw factory
Sep 18 16:16:30.066510 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.066433    2563 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:16:30.066510 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.066459    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:16:30.067602 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.067548    2563 manager.go:349] Starting recovery of all containers
Sep 18 16:16:30.094043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.093879    2563 manager.go:354] Recovery completed
Sep 18 16:16:30.096759 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.096735    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:30.098008 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.097992    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.098928 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.097939    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d6767f11b57435  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-18 16:16:30.056379445 +0000 UTC m=+2.056640246,LastTimestamp:2026-09-18 16:16:30.056379445 +0000 UTC m=+2.056640246,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}"
Sep 18 16:16:30.099883 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.099864    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.099962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.099895    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.099962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.099908    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.100427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.100414    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:16:30.100427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.100424    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.100503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.100425    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:16:30.100503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.100461    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:30.101456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.101445    2563 policy_none.go:49] "None policy: Start"
Sep 18 16:16:30.101492 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.101460    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:16:30.101492 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.101469    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:16:30.102518 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.102505    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.102571 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.102527    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.102571 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.102538    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.102724 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.102713    2563 policy_none.go:47] "Start"
Sep 18 16:16:30.110666 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.110596    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d6767f144d38b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-18 16:16:30.099880119 +0000 UTC m=+2.100140920,LastTimestamp:2026-09-18 16:16:30.099880119 +0000 UTC m=+2.100140920,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}"
Sep 18 16:16:30.119366 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.119282    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d6767f144d8e98  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-18 16:16:30.099902104 +0000 UTC m=+2.100162907,LastTimestamp:2026-09-18 16:16:30.099902104 +0000 UTC m=+2.100162907,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}"
Sep 18 16:16:30.136269 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136254    2563 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.136303    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136314    2563 server.go:87] "Starting device plugin registration server"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136544    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136581    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136681    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136752    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.136760    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.137298    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.137337    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.141985    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q9w72"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.142813    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d6767f144db3b8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-18 16:16:30.099911608 +0000 UTC m=+2.100172410,LastTimestamp:2026-09-18 16:16:30.099911608 +0000 UTC m=+2.100172410,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}"
Sep 18 16:16:30.151488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.149949    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q9w72"
Sep 18 16:16:30.151943 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.151509    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-96.ec2.internal.18d6767f144d38b7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d6767f144d38b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-18 16:16:30.099880119 +0000 UTC m=+2.100140920,LastTimestamp:2026-09-18 16:16:30.102516986 +0000 UTC m=+2.102777787,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}"
Sep 18 16:16:30.210277 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.210239    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:16:30.211767 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.211741    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:16:30.211871 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.211795    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:16:30.211871 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.211821    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:16:30.211950 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.211867    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:16:30.214291 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.214269    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:30.237438 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.237422    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.238218 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.238201    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.238300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.238233    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.238300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.238243    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.238300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.238266    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.244135 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.244122    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.244179 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.244142    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-96.ec2.internal\": node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.261865 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.261819    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.312363 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.312338    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"]
Sep 18 16:16:30.312462 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.312397    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.313307 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.313292    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.313378 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.313319    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.313378 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.313330    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.314549 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.314538    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.314665 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.314650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.314708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.314677    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.315245 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.315222    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.315320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.315251    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.315320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.315259    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.315320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.315284    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.315320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.315302    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.315320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.315313    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.316446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.316430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.316508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.316456    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:30.317184 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.317168    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:30.317252 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.317197    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:30.317252 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.317210    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:30.346082 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.346062    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.ec2.internal\" not found" node="ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.350240 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.350227    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.ec2.internal\" not found" node="ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.362847 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.362833    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.368164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.368147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6725ae6ec8ccf32198674d289bfdc598-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"6725ae6ec8ccf32198674d289bfdc598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.368238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.368178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6725ae6ec8ccf32198674d289bfdc598-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"6725ae6ec8ccf32198674d289bfdc598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.368238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.368205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e6cf529647eabbf917a54df21705ba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"d6e6cf529647eabbf917a54df21705ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.462979 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.462936    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.469323 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.469308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6725ae6ec8ccf32198674d289bfdc598-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"6725ae6ec8ccf32198674d289bfdc598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.469375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.469333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6725ae6ec8ccf32198674d289bfdc598-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"6725ae6ec8ccf32198674d289bfdc598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.469375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.469352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e6cf529647eabbf917a54df21705ba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"d6e6cf529647eabbf917a54df21705ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.469437 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.469391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6e6cf529647eabbf917a54df21705ba-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"d6e6cf529647eabbf917a54df21705ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.469475 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.469425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6725ae6ec8ccf32198674d289bfdc598-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"6725ae6ec8ccf32198674d289bfdc598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.469475 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.469425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6725ae6ec8ccf32198674d289bfdc598-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"6725ae6ec8ccf32198674d289bfdc598\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.563826 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.563753    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.649229 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.649193    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.653896 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.653369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:30.664224 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.664204    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.764671 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.764620    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.865048 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.864970    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:30.931620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:30.931591    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:30.965312 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:30.965282    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:31.019643 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.019612    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:16:31.020066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.019766    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:31.020066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.019781    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:31.065523 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:31.065497    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:31.152673 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.152608    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:31.153668 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.153648    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:31.153715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.153687    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:31.153715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.153699    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:31.154040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.154015    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:11:30 +0000 UTC" deadline="2028-07-05 22:23:03.789177068 +0000 UTC"
Sep 18 16:16:31.154082 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.154040    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15750h6m32.63513925s"
Sep 18 16:16:31.166229 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:31.166208    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:31.266632 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:31.266600    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found"
Sep 18 16:16:31.290423 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.290404    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:31.330581 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:31.330537    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e6cf529647eabbf917a54df21705ba.slice/crio-00fe82019310ce46b111bc134037a391c25d98997c1534115a19ab497d8cec10 WatchSource:0}: Error finding container 00fe82019310ce46b111bc134037a391c25d98997c1534115a19ab497d8cec10: Status 404 returned error can't find the container with id 00fe82019310ce46b111bc134037a391c25d98997c1534115a19ab497d8cec10
Sep 18 16:16:31.330910 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:31.330893    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6725ae6ec8ccf32198674d289bfdc598.slice/crio-b9f5df0d69dd8cfb5b31bb142fc31c659cdd92afe75fb79c68d3d070e90a7dcb WatchSource:0}: Error finding container b9f5df0d69dd8cfb5b31bb142fc31c659cdd92afe75fb79c68d3d070e90a7dcb: Status 404 returned error can't find the container with id b9f5df0d69dd8cfb5b31bb142fc31c659cdd92afe75fb79c68d3d070e90a7dcb
Sep 18 16:16:31.336164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.336148    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:16:31.364831 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.364800    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:31.375654 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.375633    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:31.376688 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.376677    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"
Sep 18 16:16:31.390825 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.390803    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:31.596135 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:31.596107    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:32.048121 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.048087    2563 apiserver.go:52] "Watching apiserver"
Sep 18 16:16:32.055006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.054978    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:16:32.055439 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.055414    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65","openshift-dns/node-resolver-lmmfb","openshift-image-registry/node-ca-bxdtm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal","openshift-multus/multus-h2nqg","openshift-multus/network-metrics-daemon-7t76l","openshift-network-operator/iptables-alerter-s845k","kube-system/global-pull-secret-syncer-5ws42","kube-system/konnectivity-agent-wkrr6","openshift-cluster-node-tuning-operator/tuned-rbvbv","openshift-multus/multus-additional-cni-plugins-ndq2c","openshift-network-diagnostics/network-check-target-bq84s","openshift-ovn-kubernetes/ovnkube-node-fw7m5"]
Sep 18 16:16:32.057194 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.057174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.057296 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.057252    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:32.058289 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.058270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.060009 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.059992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.060962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.060814    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j27z2\""
Sep 18 16:16:32.060962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.060822    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.061099 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.061070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.061151 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.061122    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.062321 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.062299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jsnfb\""
Sep 18 16:16:32.062419 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.062351    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.062611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.062593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.062858 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.062840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.063072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.063050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.063271 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.063255    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:16:32.063555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.063535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.063872 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.063854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j44km\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.064672    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.065393    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.065629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.066153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wdlx6\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.066323    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.066446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.066608    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.067054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.066789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:16:32.075266 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.075246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/312dfd92-b94e-484d-821a-f9ffba57a32f-serviceca\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-systemd\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-var-lib-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpj9f\" (UniqueName: \"kubernetes.io/projected/29305122-d419-448f-9807-b92cc8861913-kube-api-access-kpj9f\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-etc-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29305122-d419-448f-9807-b92cc8861913-tmp-dir\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.076978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/312dfd92-b94e-484d-821a-f9ffba57a32f-host\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5njb\" (UniqueName: \"kubernetes.io/projected/312dfd92-b94e-484d-821a-f9ffba57a32f-kube-api-access-c5njb\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.077000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-kubelet\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-systemd-units\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-run-netns\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-ovn\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-node-log\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.077407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr6z\" (UniqueName: \"kubernetes.io/projected/66a797a2-9e7c-4c77-9523-2e62b0cccb79-kube-api-access-bwr6z\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.078217 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29305122-d419-448f-9807-b92cc8861913-hosts-file\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.078217 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-slash\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.078217 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.077212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.078361 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.078347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cvtk6\""
Sep 18 16:16:32.078421 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.078360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:16:32.078421 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.078379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.078621 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.078601    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.078770 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.078757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:16:32.079620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.079602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:32.079693 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.079675    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:32.081695 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.081676    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:32.082659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.082644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.083999 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.083983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.085418 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.085399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.085799 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.085777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:16:32.086043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086021    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.086119 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.086174 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086135    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:16:32.086368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:16:32.086498 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t764l\""
Sep 18 16:16:32.086659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.087059 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.086554    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7wqf\""
Sep 18 16:16:32.087654 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.087636    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:32.087905 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.087882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:32.087994 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.087882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:16:32.088100 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.088082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4q26\""
Sep 18 16:16:32.088944 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.088928    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:16:32.089059 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.089038    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:16:32.089139 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.089071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pwtd9\""
Sep 18 16:16:32.102356 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.102325    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9xchj"
Sep 18 16:16:32.109180 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.109163    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9xchj"
Sep 18 16:16:32.165819 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.165788    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:16:32.178220 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysconfig\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.178220 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-lib-modules\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-device-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-var-lib-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-netns\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-sys\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-var-lib-kubelet\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovn-node-metrics-cert\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.178417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w2h5\" (UniqueName: \"kubernetes.io/projected/64a4bc63-9c28-4830-81d2-8d2dd866b848-kube-api-access-9w2h5\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dd636e67-a256-4028-b55d-7dbf12f724f7-kubelet-config\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/798956e5-559d-4944-b555-f186abf61335-agent-certs\") pod \"konnectivity-agent-wkrr6\" (UID: \"798956e5-559d-4944-b555-f186abf61335\") " pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsz9k\" (UniqueName: \"kubernetes.io/projected/e5841404-0c91-44f3-9153-111da58f2daa-kube-api-access-zsz9k\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/312dfd92-b94e-484d-821a-f9ffba57a32f-host\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5njb\" (UniqueName: \"kubernetes.io/projected/312dfd92-b94e-484d-821a-f9ffba57a32f-kube-api-access-c5njb\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-var-lib-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-systemd-units\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-node-log\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-log-socket\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-system-cni-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/312dfd92-b94e-484d-821a-f9ffba57a32f-host\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5841404-0c91-44f3-9153-111da58f2daa-multus-daemon-config\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-systemd-units\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-multus-certs\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-node-log\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.179867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysctl-conf\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-tuned\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/312dfd92-b94e-484d-821a-f9ffba57a32f-serviceca\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.178872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-systemd\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-kubelet\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-modprobe-d\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hddh9\" (UniqueName: \"kubernetes.io/projected/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-kube-api-access-hddh9\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-system-cni-dir\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-systemd\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-etc-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/312dfd92-b94e-484d-821a-f9ffba57a32f-serviceca\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovnkube-script-lib\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-etc-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-run\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/798956e5-559d-4944-b555-f186abf61335-konnectivity-ca\") pod \"konnectivity-agent-wkrr6\" (UID: \"798956e5-559d-4944-b555-f186abf61335\") " pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/132532f8-4679-4c8c-b617-1b2ac984951a-host-slash\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.180690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-run-ovn-kubernetes\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-kubelet\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-ovn\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-kubelet\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-k8s-cni-cncf-io\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-ovn\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysctl-d\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpfx\" (UniqueName: \"kubernetes.io/projected/981024a2-6f8a-47d8-b8fd-76833fed016b-kube-api-access-8kpfx\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr6z\" (UniqueName: \"kubernetes.io/projected/66a797a2-9e7c-4c77-9523-2e62b0cccb79-kube-api-access-bwr6z\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.179845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-slash\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovnkube-config\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-etc-kubernetes\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-socket-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-cnibin\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-socket-dir-parent\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.181582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-kubernetes\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-sys-fs\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-cni-netd\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29305122-d419-448f-9807-b92cc8861913-tmp-dir\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-run-netns\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-cni-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-slash\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-hostroot\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-host\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cni-binary-copy\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-env-overrides\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-registration-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-systemd\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.182221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5841404-0c91-44f3-9153-111da58f2daa-cni-binary-copy\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29305122-d419-448f-9807-b92cc8861913-tmp-dir\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-cni-bin\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981024a2-6f8a-47d8-b8fd-76833fed016b-tmp\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cnibin\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-run-openvswitch\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-run-netns\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.180979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpj9f\" (UniqueName: \"kubernetes.io/projected/29305122-d419-448f-9807-b92cc8861913-kube-api-access-kpj9f\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-conf-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.180982    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-cni-bin\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.181161    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:32.681139527 +0000 UTC m=+4.681400333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/132532f8-4679-4c8c-b617-1b2ac984951a-iptables-alerter-script\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbq8k\" (UniqueName: \"kubernetes.io/projected/132532f8-4679-4c8c-b617-1b2ac984951a-kube-api-access-nbq8k\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdn8s\" (UniqueName: \"kubernetes.io/projected/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-kube-api-access-xdn8s\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.182821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29305122-d419-448f-9807-b92cc8861913-hosts-file\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.183316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-cni-multus\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.183316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dd636e67-a256-4028-b55d-7dbf12f724f7-dbus\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.183316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.183316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.181658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/29305122-d419-448f-9807-b92cc8861913-hosts-file\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.186049 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.186018    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:16:32.190027 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.190004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5njb\" (UniqueName: \"kubernetes.io/projected/312dfd92-b94e-484d-821a-f9ffba57a32f-kube-api-access-c5njb\") pod \"node-ca-bxdtm\" (UID: \"312dfd92-b94e-484d-821a-f9ffba57a32f\") " pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.190136 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.190005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr6z\" (UniqueName: \"kubernetes.io/projected/66a797a2-9e7c-4c77-9523-2e62b0cccb79-kube-api-access-bwr6z\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.190473 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.190412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpj9f\" (UniqueName: \"kubernetes.io/projected/29305122-d419-448f-9807-b92cc8861913-kube-api-access-kpj9f\") pod \"node-resolver-lmmfb\" (UID: \"29305122-d419-448f-9807-b92cc8861913\") " pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.221220 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.221161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" event={"ID":"d6e6cf529647eabbf917a54df21705ba","Type":"ContainerStarted","Data":"00fe82019310ce46b111bc134037a391c25d98997c1534115a19ab497d8cec10"}
Sep 18 16:16:32.223088 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.223059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"6725ae6ec8ccf32198674d289bfdc598","Type":"ContainerStarted","Data":"b9f5df0d69dd8cfb5b31bb142fc31c659cdd92afe75fb79c68d3d070e90a7dcb"}
Sep 18 16:16:32.282151 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-kubelet\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.282151 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-modprobe-d\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hddh9\" (UniqueName: \"kubernetes.io/projected/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-kube-api-access-hddh9\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-system-cni-dir\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovnkube-script-lib\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-run\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/798956e5-559d-4944-b555-f186abf61335-konnectivity-ca\") pod \"konnectivity-agent-wkrr6\" (UID: \"798956e5-559d-4944-b555-f186abf61335\") " pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/132532f8-4679-4c8c-b617-1b2ac984951a-host-slash\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-kubelet\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.282370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-run\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-run-ovn-kubernetes\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-k8s-cni-cncf-io\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysctl-d\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpfx\" (UniqueName: \"kubernetes.io/projected/981024a2-6f8a-47d8-b8fd-76833fed016b-kube-api-access-8kpfx\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovnkube-config\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-etc-kubernetes\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-socket-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-cnibin\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-socket-dir-parent\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-kubernetes\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.282728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-sys-fs\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-cni-netd\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-cni-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-hostroot\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-host\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cni-binary-copy\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-env-overrides\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-registration-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-systemd\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.283164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-systemd\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/798956e5-559d-4944-b555-f186abf61335-konnectivity-ca\") pod \"konnectivity-agent-wkrr6\" (UID: \"798956e5-559d-4944-b555-f186abf61335\") " pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-system-cni-dir\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5841404-0c91-44f3-9153-111da58f2daa-cni-binary-copy\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-cni-bin\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981024a2-6f8a-47d8-b8fd-76833fed016b-tmp\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cnibin\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-conf-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-cni-bin\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/132532f8-4679-4c8c-b617-1b2ac984951a-iptables-alerter-script\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbq8k\" (UniqueName: \"kubernetes.io/projected/132532f8-4679-4c8c-b617-1b2ac984951a-kube-api-access-nbq8k\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn8s\" (UniqueName: \"kubernetes.io/projected/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-kube-api-access-xdn8s\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-cni-multus\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-run-ovn-kubernetes\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.283548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dd636e67-a256-4028-b55d-7dbf12f724f7-dbus\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysconfig\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dd636e67-a256-4028-b55d-7dbf12f724f7-dbus\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-lib-modules\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/132532f8-4679-4c8c-b617-1b2ac984951a-host-slash\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-device-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-netns\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-sys\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-var-lib-kubelet\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovn-node-metrics-cert\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w2h5\" (UniqueName: \"kubernetes.io/projected/64a4bc63-9c28-4830-81d2-8d2dd866b848-kube-api-access-9w2h5\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dd636e67-a256-4028-b55d-7dbf12f724f7-kubelet-config\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovnkube-script-lib\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/798956e5-559d-4944-b555-f186abf61335-agent-certs\") pod \"konnectivity-agent-wkrr6\" (UID: \"798956e5-559d-4944-b555-f186abf61335\") " pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsz9k\" (UniqueName: \"kubernetes.io/projected/e5841404-0c91-44f3-9153-111da58f2daa-kube-api-access-zsz9k\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-kubernetes\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-log-socket\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-system-cni-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-sys-fs\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5841404-0c91-44f3-9153-111da58f2daa-multus-daemon-config\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-cni-netd\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-multus-certs\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysctl-conf\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-tuned\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5841404-0c91-44f3-9153-111da58f2daa-cni-binary-copy\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-cni-bin\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5841404-0c91-44f3-9153-111da58f2daa-multus-daemon-config\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-k8s-cni-cncf-io\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysctl-d\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-netns\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-sys\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.284681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-socket-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.284824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-cnibin\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.284893    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.284956    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:32.784938542 +0000 UTC m=+4.785199335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-var-lib-kubelet\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-env-overrides\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysconfig\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-lib-modules\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.282317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-modprobe-d\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-device-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.285508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovnkube-config\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.286067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-etc-kubernetes\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.286067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-cni-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.286067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-hostroot\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.286067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.285711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-host\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.286255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/132532f8-4679-4c8c-b617-1b2ac984951a-iptables-alerter-script\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.286255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cni-binary-copy\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.286255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-cnibin\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.286487 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dd636e67-a256-4028-b55d-7dbf12f724f7-kubelet-config\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.286608 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-conf-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.286678 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.286736 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.286736 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-host-cni-bin\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.286839 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64a4bc63-9c28-4830-81d2-8d2dd866b848-log-socket\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.286839 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-registration-dir\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.286839 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.283447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-multus-socket-dir-parent\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.286839 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-tuned\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.287006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-var-lib-cni-multus\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.287006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-system-cni-dir\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.287006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.287006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/981024a2-6f8a-47d8-b8fd-76833fed016b-etc-sysctl-conf\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.287006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.286935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5841404-0c91-44f3-9153-111da58f2daa-host-run-multus-certs\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.287514 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.287478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64a4bc63-9c28-4830-81d2-8d2dd866b848-ovn-node-metrics-cert\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.287687 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.287671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981024a2-6f8a-47d8-b8fd-76833fed016b-tmp\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.289174 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.289150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/798956e5-559d-4944-b555-f186abf61335-agent-certs\") pod \"konnectivity-agent-wkrr6\" (UID: \"798956e5-559d-4944-b555-f186abf61335\") " pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.291535 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.291488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hddh9\" (UniqueName: \"kubernetes.io/projected/6df09439-2fc2-48bb-bce2-7bf0e98e92cd-kube-api-access-hddh9\") pod \"aws-ebs-csi-driver-node-zzp65\" (UID: \"6df09439-2fc2-48bb-bce2-7bf0e98e92cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.291987 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.291967    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:32.292477 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.291993    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:32.292477 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.292008    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:32.292477 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.292084    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:32.792064974 +0000 UTC m=+4.792325781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:32.293019 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.292995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpfx\" (UniqueName: \"kubernetes.io/projected/981024a2-6f8a-47d8-b8fd-76833fed016b-kube-api-access-8kpfx\") pod \"tuned-rbvbv\" (UID: \"981024a2-6f8a-47d8-b8fd-76833fed016b\") " pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.294187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.294150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsz9k\" (UniqueName: \"kubernetes.io/projected/e5841404-0c91-44f3-9153-111da58f2daa-kube-api-access-zsz9k\") pod \"multus-h2nqg\" (UID: \"e5841404-0c91-44f3-9153-111da58f2daa\") " pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.294699 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.294678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbq8k\" (UniqueName: \"kubernetes.io/projected/132532f8-4679-4c8c-b617-1b2ac984951a-kube-api-access-nbq8k\") pod \"iptables-alerter-s845k\" (UID: \"132532f8-4679-4c8c-b617-1b2ac984951a\") " pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.295488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.295469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn8s\" (UniqueName: \"kubernetes.io/projected/8746475b-5bd4-4eb0-b54a-93ebca3ba19f-kube-api-access-xdn8s\") pod \"multus-additional-cni-plugins-ndq2c\" (UID: \"8746475b-5bd4-4eb0-b54a-93ebca3ba19f\") " pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.296760 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.296740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w2h5\" (UniqueName: \"kubernetes.io/projected/64a4bc63-9c28-4830-81d2-8d2dd866b848-kube-api-access-9w2h5\") pod \"ovnkube-node-fw7m5\" (UID: \"64a4bc63-9c28-4830-81d2-8d2dd866b848\") " pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.331916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.331666    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:32.379730 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.379697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rbvbv"
Sep 18 16:16:32.387971 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.387948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lmmfb"
Sep 18 16:16:32.401960 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.401938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bxdtm"
Sep 18 16:16:32.405022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.404997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:32.411832 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.411812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h2nqg"
Sep 18 16:16:32.420691 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.420671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65"
Sep 18 16:16:32.428420 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.428403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:32.437474 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.437458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s845k"
Sep 18 16:16:32.444853 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.444834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ndq2c"
Sep 18 16:16:32.687585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.687477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:32.687754 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.687639    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:32.687754 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.687706    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:33.687690228 +0000 UTC m=+5.687951018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:32.788812 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.788777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:32.788962 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.788912    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:32.789013 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.788975    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:33.78896017 +0000 UTC m=+5.789220964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:32.889368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:32.889321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:32.889542 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.889524    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:32.889621 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.889553    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:32.889621 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.889582    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:32.889725 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:32.889648    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:33.88963063 +0000 UTC m=+5.889891444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:33.111357 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.111267    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:32 +0000 UTC" deadline="2028-04-02 20:21:15.379207451 +0000 UTC"
Sep 18 16:16:33.111357 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.111300    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13492h4m42.267912263s"
Sep 18 16:16:33.213050 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.213009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:33.213220 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.213131    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:33.213220 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.213177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:33.213325 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.213289    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:33.346446 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.346423    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29305122_d419_448f_9807_b92cc8861913.slice/crio-c8f01941d76eb39a2e7e160425a808627935b2a78e00718ff0095a67956902ba WatchSource:0}: Error finding container c8f01941d76eb39a2e7e160425a808627935b2a78e00718ff0095a67956902ba: Status 404 returned error can't find the container with id c8f01941d76eb39a2e7e160425a808627935b2a78e00718ff0095a67956902ba
Sep 18 16:16:33.347877 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.347748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5841404_0c91_44f3_9153_111da58f2daa.slice/crio-db77ef506a584a7b5b36f25bfa1b705952d811081139c76a43a8c210a0c14152 WatchSource:0}: Error finding container db77ef506a584a7b5b36f25bfa1b705952d811081139c76a43a8c210a0c14152: Status 404 returned error can't find the container with id db77ef506a584a7b5b36f25bfa1b705952d811081139c76a43a8c210a0c14152
Sep 18 16:16:33.349636 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.348612    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132532f8_4679_4c8c_b617_1b2ac984951a.slice/crio-b1838d22244988cfe16238998691c68a403960121cd6d421da3b5f5523662b2f WatchSource:0}: Error finding container b1838d22244988cfe16238998691c68a403960121cd6d421da3b5f5523662b2f: Status 404 returned error can't find the container with id b1838d22244988cfe16238998691c68a403960121cd6d421da3b5f5523662b2f
Sep 18 16:16:33.352382 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.352360    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8746475b_5bd4_4eb0_b54a_93ebca3ba19f.slice/crio-9a68b087c3b9e9f9cf9a563df81cc193ddd08aad1dd2863eaa3941775c2dae4c WatchSource:0}: Error finding container 9a68b087c3b9e9f9cf9a563df81cc193ddd08aad1dd2863eaa3941775c2dae4c: Status 404 returned error can't find the container with id 9a68b087c3b9e9f9cf9a563df81cc193ddd08aad1dd2863eaa3941775c2dae4c
Sep 18 16:16:33.353723 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.353698    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798956e5_559d_4944_b555_f186abf61335.slice/crio-b218d86dc07e3f360d4a939a990b3e2ee55bee0591ff46f9cd9b0fb9275de94c WatchSource:0}: Error finding container b218d86dc07e3f360d4a939a990b3e2ee55bee0591ff46f9cd9b0fb9275de94c: Status 404 returned error can't find the container with id b218d86dc07e3f360d4a939a990b3e2ee55bee0591ff46f9cd9b0fb9275de94c
Sep 18 16:16:33.354932 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.354890    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981024a2_6f8a_47d8_b8fd_76833fed016b.slice/crio-a02d7ccc49304237cdf270a48afbc5c6be5d66bd0402c46ea42d028bf5f5dd28 WatchSource:0}: Error finding container a02d7ccc49304237cdf270a48afbc5c6be5d66bd0402c46ea42d028bf5f5dd28: Status 404 returned error can't find the container with id a02d7ccc49304237cdf270a48afbc5c6be5d66bd0402c46ea42d028bf5f5dd28
Sep 18 16:16:33.356254 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.355819    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df09439_2fc2_48bb_bce2_7bf0e98e92cd.slice/crio-22c5e72a0385206ecc3823fb8d0032ca83a83768b4ab9223442ad1b7dcd49644 WatchSource:0}: Error finding container 22c5e72a0385206ecc3823fb8d0032ca83a83768b4ab9223442ad1b7dcd49644: Status 404 returned error can't find the container with id 22c5e72a0385206ecc3823fb8d0032ca83a83768b4ab9223442ad1b7dcd49644
Sep 18 16:16:33.356489 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.356358    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312dfd92_b94e_484d_821a_f9ffba57a32f.slice/crio-4b48724fde18bf87cb629285357da6cb7f7c51fdac740aed458fbaa849e90a06 WatchSource:0}: Error finding container 4b48724fde18bf87cb629285357da6cb7f7c51fdac740aed458fbaa849e90a06: Status 404 returned error can't find the container with id 4b48724fde18bf87cb629285357da6cb7f7c51fdac740aed458fbaa849e90a06
Sep 18 16:16:33.359981 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:16:33.358765    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a4bc63_9c28_4830_81d2_8d2dd866b848.slice/crio-4e7125e95186343d07c1f1597f465e37542f910649d1a50f31b6614977ea260a WatchSource:0}: Error finding container 4e7125e95186343d07c1f1597f465e37542f910649d1a50f31b6614977ea260a: Status 404 returned error can't find the container with id 4e7125e95186343d07c1f1597f465e37542f910649d1a50f31b6614977ea260a
Sep 18 16:16:33.635357 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.635264    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:33.695461 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.695416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:33.695648 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.695614    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:33.695713 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.695679    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:35.695660586 +0000 UTC m=+7.695921399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:33.796190 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.796154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:33.796361 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.796310    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:33.796417 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.796375    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:35.796357256 +0000 UTC m=+7.796618058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:33.896864 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:33.896781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:33.897025 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.897001    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:33.897025 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.897021    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:33.897131 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.897035    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:33.897131 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:33.897092    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:35.897074402 +0000 UTC m=+7.897335207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:34.112335 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.112255    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:32 +0000 UTC" deadline="2028-04-17 22:28:41.86146958 +0000 UTC"
Sep 18 16:16:34.112335 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.112307    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13854h12m7.749165289s"
Sep 18 16:16:34.215376 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.214854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:34.215376 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:34.214973    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:34.237169 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.236460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" event={"ID":"d6e6cf529647eabbf917a54df21705ba","Type":"ContainerStarted","Data":"1225bc6676e06724c5642450e31eed54bc38e9a40632156dbcd7479e734b869b"}
Sep 18 16:16:34.239210 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.239108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"4e7125e95186343d07c1f1597f465e37542f910649d1a50f31b6614977ea260a"}
Sep 18 16:16:34.247838 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.247749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bxdtm" event={"ID":"312dfd92-b94e-484d-821a-f9ffba57a32f","Type":"ContainerStarted","Data":"4b48724fde18bf87cb629285357da6cb7f7c51fdac740aed458fbaa849e90a06"}
Sep 18 16:16:34.257135 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.257010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65" event={"ID":"6df09439-2fc2-48bb-bce2-7bf0e98e92cd","Type":"ContainerStarted","Data":"22c5e72a0385206ecc3823fb8d0032ca83a83768b4ab9223442ad1b7dcd49644"}
Sep 18 16:16:34.259395 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.259323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lmmfb" event={"ID":"29305122-d419-448f-9807-b92cc8861913","Type":"ContainerStarted","Data":"c8f01941d76eb39a2e7e160425a808627935b2a78e00718ff0095a67956902ba"}
Sep 18 16:16:34.267585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.267136    2563 generic.go:356] "Generic (PLEG): container finished" podID="6725ae6ec8ccf32198674d289bfdc598" containerID="300bcf0924c62a89ea11d50a4bfdb7d9a1e7aa999fef097b7e5886de75660b89" exitCode=0
Sep 18 16:16:34.267585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.267208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"6725ae6ec8ccf32198674d289bfdc598","Type":"ContainerDied","Data":"300bcf0924c62a89ea11d50a4bfdb7d9a1e7aa999fef097b7e5886de75660b89"}
Sep 18 16:16:34.274167 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.274104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rbvbv" event={"ID":"981024a2-6f8a-47d8-b8fd-76833fed016b","Type":"ContainerStarted","Data":"a02d7ccc49304237cdf270a48afbc5c6be5d66bd0402c46ea42d028bf5f5dd28"}
Sep 18 16:16:34.279634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.279542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wkrr6" event={"ID":"798956e5-559d-4944-b555-f186abf61335","Type":"ContainerStarted","Data":"b218d86dc07e3f360d4a939a990b3e2ee55bee0591ff46f9cd9b0fb9275de94c"}
Sep 18 16:16:34.284419 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.283657    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" podStartSLOduration=3.28363937 podStartE2EDuration="3.28363937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:34.252047148 +0000 UTC m=+6.252307974" watchObservedRunningTime="2026-09-18 16:16:34.28363937 +0000 UTC m=+6.283900183"
Sep 18 16:16:34.293442 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.293411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerStarted","Data":"9a68b087c3b9e9f9cf9a563df81cc193ddd08aad1dd2863eaa3941775c2dae4c"}
Sep 18 16:16:34.302951 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.302877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s845k" event={"ID":"132532f8-4679-4c8c-b617-1b2ac984951a","Type":"ContainerStarted","Data":"b1838d22244988cfe16238998691c68a403960121cd6d421da3b5f5523662b2f"}
Sep 18 16:16:34.317664 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:34.317633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h2nqg" event={"ID":"e5841404-0c91-44f3-9153-111da58f2daa","Type":"ContainerStarted","Data":"db77ef506a584a7b5b36f25bfa1b705952d811081139c76a43a8c210a0c14152"}
Sep 18 16:16:35.212474 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:35.212441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:35.212951 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:35.212641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:35.212951 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.212724    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:35.212951 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.212634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:35.346461 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:35.346426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"6725ae6ec8ccf32198674d289bfdc598","Type":"ContainerStarted","Data":"aee9a2aaca00c5ca313eecfe9a72d356715268137066a5c6e49d46870638aaa1"}
Sep 18 16:16:35.711117 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:35.710423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:35.711117 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.710637    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:35.711117 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.710698    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:39.710679372 +0000 UTC m=+11.710940181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:35.810995 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:35.810888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:35.811178 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.811055    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:35.811178 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.811115    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:39.811097163 +0000 UTC m=+11.811357953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:35.911585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:35.911536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:35.911813 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.911794    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:35.911895 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.911822    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:35.911895 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.911835    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:35.911895 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:35.911890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:39.911873197 +0000 UTC m=+11.912133989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:36.213686 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:36.213662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:36.214128 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:36.213751    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:37.212591 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:37.212514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:37.212763 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:37.212644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:37.212954 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:37.212937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:37.213032 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:37.213017    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:38.215173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:38.214692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:38.215173 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:38.214818    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:39.212427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:39.212391    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:39.212621 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.212523    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:39.212987 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:39.212962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:39.213104 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.213061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:39.743666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:39.743097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:39.743666 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.743235    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:39.743666 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.743306    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:47.743287707 +0000 UTC m=+19.743548499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:39.844153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:39.844069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:39.844330 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.844247    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:39.844330 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.844321    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:47.844301192 +0000 UTC m=+19.844561997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:39.944636 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:39.944587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:39.944859 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.944787    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:39.944859 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.944814    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:39.944859 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.944827    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:39.945004 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:39.944888    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:16:47.944869338 +0000 UTC m=+19.945130140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:40.214756 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:40.214661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:40.216042 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:40.215737    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:41.212216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:41.212183    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:41.212612 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:41.212307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:41.212750 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:41.212728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:41.212842 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:41.212827    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:42.213729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:42.213689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:42.214212 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:42.213832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:43.212502 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:43.212464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:43.212502 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:43.212498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:43.212740 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:43.212601    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:43.212778 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:43.212742    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:44.212299 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:44.212265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:44.212752 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:44.212396    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:45.212692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:45.212662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:45.213097 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:45.212774    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:45.213155 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:45.213105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:45.213229 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:45.213199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:46.212184 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:46.212145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:46.212371 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:46.212272    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:47.212298 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:47.212272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:47.212673 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:47.212362    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:47.212673 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:47.212414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:47.212673 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:47.212501    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:47.806816 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:47.806770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:47.806992 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:47.806947    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:47.807080 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:47.807013    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:03.806993078 +0000 UTC m=+35.807253876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:47.907946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:47.907903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:47.908109 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:47.908054    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:47.908176 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:47.908141    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:03.90811268 +0000 UTC m=+35.908373474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:48.008945 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:48.008904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:48.009116 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:48.009060    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:48.009116 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:48.009086    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:48.009116 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:48.009099    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:48.009248 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:48.009161    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:17:04.009143461 +0000 UTC m=+36.009404267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:48.212805 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:48.212768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:48.213242 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:48.212890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:49.212173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:49.212133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:49.212350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:49.212133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:49.212350 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:49.212273    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:49.212472 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:49.212363    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:50.216379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:50.216342    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:50.216860 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:50.216458    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:51.212990 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.212830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:51.213190 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:51.213061    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:51.213190 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.212880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:51.213190 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:51.213164    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:51.381422 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.381164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rbvbv" event={"ID":"981024a2-6f8a-47d8-b8fd-76833fed016b","Type":"ContainerStarted","Data":"68c15c10fb99a1ba7a689cf5e42c84afefa0344c14859b19c603f3ff4675c236"}
Sep 18 16:16:51.382712 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.382687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wkrr6" event={"ID":"798956e5-559d-4944-b555-f186abf61335","Type":"ContainerStarted","Data":"e1fae4084fd76eda33d80913868c7a21f54faeaebf9d4b7fad989c621998fd2f"}
Sep 18 16:16:51.386751 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.386665    2563 generic.go:356] "Generic (PLEG): container finished" podID="8746475b-5bd4-4eb0-b54a-93ebca3ba19f" containerID="98c4ef8486cd2d1ff0db9e938a0ce21c489ff415bad5c078034225f48091a8d2" exitCode=0
Sep 18 16:16:51.386751 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.386739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerDied","Data":"98c4ef8486cd2d1ff0db9e938a0ce21c489ff415bad5c078034225f48091a8d2"}
Sep 18 16:16:51.388338 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.388313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h2nqg" event={"ID":"e5841404-0c91-44f3-9153-111da58f2daa","Type":"ContainerStarted","Data":"e9fa4b7cb276715291c82dd77c209ce7627483dd261f8fbb4a2c6a076563d61e"}
Sep 18 16:16:51.390256 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.390241    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:16:51.390647 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.390627    2563 generic.go:356] "Generic (PLEG): container finished" podID="64a4bc63-9c28-4830-81d2-8d2dd866b848" containerID="5281e7ad066524846624d293e493436bffa60422829157495eac1aee75c3bb1c" exitCode=1
Sep 18 16:16:51.390723 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.390691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"a3986e1e0be7c96bee1ce8b809182ffa9682e83bf36a735a1e2c73ecdf97c611"}
Sep 18 16:16:51.390782 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.390766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerDied","Data":"5281e7ad066524846624d293e493436bffa60422829157495eac1aee75c3bb1c"}
Sep 18 16:16:51.390845 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.390825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"a85ea39c757737902fe5c0e4fa8fe96909881e720d26230f7c4016f49f439f9b"}
Sep 18 16:16:51.392152 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.392127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bxdtm" event={"ID":"312dfd92-b94e-484d-821a-f9ffba57a32f","Type":"ContainerStarted","Data":"14a40a321fd9806f6eedfbbe7a2d7c198534e663709429f19e91ab11b01b684f"}
Sep 18 16:16:51.393580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.393493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65" event={"ID":"6df09439-2fc2-48bb-bce2-7bf0e98e92cd","Type":"ContainerStarted","Data":"3d4004fc254c162b44ce20db0334f9430795c6671c2593491bb055697c38c2bb"}
Sep 18 16:16:51.394876 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.394854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lmmfb" event={"ID":"29305122-d419-448f-9807-b92cc8861913","Type":"ContainerStarted","Data":"c90c37a01c12ed284531c2c1e14c11b9e836bd24ca2918dab1f237ffbedcb91a"}
Sep 18 16:16:51.400723 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.400491    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" podStartSLOduration=20.400474566 podStartE2EDuration="20.400474566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:35.363362676 +0000 UTC m=+7.363623489" watchObservedRunningTime="2026-09-18 16:16:51.400474566 +0000 UTC m=+23.400735378"
Sep 18 16:16:51.414161 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.414112    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wkrr6" podStartSLOduration=3.8807273159999998 podStartE2EDuration="21.414096268s" totalImagesPullingTime="17.533368952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:51.413882 +0000 UTC m=+23.414142811" watchObservedRunningTime="2026-09-18 16:16:51.414096268 +0000 UTC m=+23.414357079"
Sep 18 16:16:51.414263 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.414214    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rbvbv" podStartSLOduration=3.883563796 podStartE2EDuration="21.414208089s" totalImagesPullingTime="17.530644293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:51.400066878 +0000 UTC m=+23.400327718" watchObservedRunningTime="2026-09-18 16:16:51.414208089 +0000 UTC m=+23.414468900"
Sep 18 16:16:51.450295 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.450251    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bxdtm" podStartSLOduration=3.926089592 podStartE2EDuration="21.450236542s" totalImagesPullingTime="17.52414695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:51.450178656 +0000 UTC m=+23.450439465" watchObservedRunningTime="2026-09-18 16:16:51.450236542 +0000 UTC m=+23.450497352"
Sep 18 16:16:51.496527 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.496491    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lmmfb" podStartSLOduration=3.9532408180000003 podStartE2EDuration="21.496476098s" totalImagesPullingTime="17.54323528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:51.49637134 +0000 UTC m=+23.496632151" watchObservedRunningTime="2026-09-18 16:16:51.496476098 +0000 UTC m=+23.496736908"
Sep 18 16:16:51.496699 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.496677    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h2nqg" podStartSLOduration=3.881448455 podStartE2EDuration="21.49666762s" totalImagesPullingTime="17.615219165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:51.478145987 +0000 UTC m=+23.478406797" watchObservedRunningTime="2026-09-18 16:16:51.49666762 +0000 UTC m=+23.496928430"
Sep 18 16:16:51.621391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.621367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:51.701058 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:51.701028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:52.213134 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.213062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:52.213312 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:52.213159    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:52.373048 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.373001    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:16:52.401077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.401056    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:16:52.401547 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.401523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"46f84fa3076dba8359abebf07c775e724d90006bd8886028f984cd1234db372f"}
Sep 18 16:16:52.401657 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.401627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"ecaefe867f3f58ad6bef72847a95ad06e7a3d4182a45afbde733fcc870515f96"}
Sep 18 16:16:52.401731 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.401679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"e649a528e9f853cd5e3a424fdd64a990c74f918ccd06902d0e89f362bd450154"}
Sep 18 16:16:52.403447 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.403420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65" event={"ID":"6df09439-2fc2-48bb-bce2-7bf0e98e92cd","Type":"ContainerStarted","Data":"a4e29d08e0dcabfd3b0eb403b6cfad1b8c7652cbc8867d97df8be4067b507b32"}
Sep 18 16:16:52.405014 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.404993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s845k" event={"ID":"132532f8-4679-4c8c-b617-1b2ac984951a","Type":"ContainerStarted","Data":"4c394c310703cf84b05b559b047ed3f302ed0a5428a950f5161cabec006b132b"}
Sep 18 16:16:52.419983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.419930    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s845k" podStartSLOduration=4.888620802 podStartE2EDuration="22.419911476s" totalImagesPullingTime="17.531290674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:52.419296821 +0000 UTC m=+24.419557648" watchObservedRunningTime="2026-09-18 16:16:52.419911476 +0000 UTC m=+24.420172302"
Sep 18 16:16:52.600177 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.600102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:52.700611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:52.700554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:53.151526 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:53.151232    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:16:52.373016585Z","UUID":"fc512829-797c-4117-8c64-ed55e5ae2619","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:16:53.152955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:53.152935    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:16:53.153071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:53.152970    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:16:53.212538 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:53.212510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:53.212538 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:53.212545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:53.212770 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:53.212646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:53.212815 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:53.212772    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:54.212522 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:54.212489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:54.212879 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:54.212617    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:54.413902 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:54.413874    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:16:54.414360 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:54.414328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"19f175b1dec89d33142a827d9c8e051f87c0e9461acd4dce43d0b2dacd211e1b"}
Sep 18 16:16:54.416780 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:54.416757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65" event={"ID":"6df09439-2fc2-48bb-bce2-7bf0e98e92cd","Type":"ContainerStarted","Data":"c56e2b961b2deb07843af49366b13c1fde4dd5ea0e4993a975d929d1632e9443"}
Sep 18 16:16:54.435830 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:54.435787    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzp65" podStartSLOduration=4.771472031 podStartE2EDuration="24.435744015s" totalImagesPullingTime="19.664271984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:54.435201697 +0000 UTC m=+26.435462509" watchObservedRunningTime="2026-09-18 16:16:54.435744015 +0000 UTC m=+26.436004818"
Sep 18 16:16:55.212889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:55.212624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:55.213373 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:55.212646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:55.213373 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:55.212949    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:55.213373 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:55.213016    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:56.126199 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.125463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:56.126772 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.126753    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:56.213166 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.212972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:56.213166 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:56.213094    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:56.428187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.428159    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:16:56.428688 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.428664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"b075409ea49534a7314e5f5a18327e32d05e130df90d513974ddfd64f4b4c1d0"}
Sep 18 16:16:56.428857 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.428835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:56.429126 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.429086    2563 scope.go:117] "RemoveContainer" containerID="5281e7ad066524846624d293e493436bffa60422829157495eac1aee75c3bb1c"
Sep 18 16:16:56.429368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.429341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wkrr6"
Sep 18 16:16:56.517545 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:56.517516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:16:57.213257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.212936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:57.213722 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.213095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:57.213722 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:57.213387    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:57.213722 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:57.213508    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:57.988815 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.988779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bq84s"]
Sep 18 16:16:57.988969 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.988951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:16:57.989081 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:57.989057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:16:57.989231 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.989206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ws42"]
Sep 18 16:16:57.989330 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.989315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:16:57.989443 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:57.989423    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:16:57.990064 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.990040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7t76l"]
Sep 18 16:16:57.990161 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:57.990149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:16:57.990282 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:16:57.990259    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:16:58.439130 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.439105    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:16:58.439623 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.439542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" event={"ID":"64a4bc63-9c28-4830-81d2-8d2dd866b848","Type":"ContainerStarted","Data":"a330219e2fc556f49ed6378b1372a460ef9deb43697ddf44f41c8b8a76215bf2"}
Sep 18 16:16:58.439623 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.439616    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:58.439761 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.439747    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:58.439814 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.439799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:58.441997 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.441972    2563 generic.go:356] "Generic (PLEG): container finished" podID="8746475b-5bd4-4eb0-b54a-93ebca3ba19f" containerID="bbdb10047d9608170a4dad1a710125cd3307e4284c99ef07ee731c5acab96916" exitCode=0
Sep 18 16:16:58.442119 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.442000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerDied","Data":"bbdb10047d9608170a4dad1a710125cd3307e4284c99ef07ee731c5acab96916"}
Sep 18 16:16:58.455684 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.455665    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:58.455766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.455759    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:16:58.474622 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:58.474584    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" podStartSLOduration=10.799959996 podStartE2EDuration="28.47455325s" totalImagesPullingTime="17.674593254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:58.474230875 +0000 UTC m=+30.474491686" watchObservedRunningTime="2026-09-18 16:16:58.47455325 +0000 UTC m=+30.474814058"
Sep 18 16:16:59.449944 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:59.449912    2563 generic.go:356] "Generic (PLEG): container finished" podID="8746475b-5bd4-4eb0-b54a-93ebca3ba19f" containerID="55de90d1b8cb652143c29305c2bc826fef22c449ec7e56f551ca26e6b6ccf13f" exitCode=0
Sep 18 16:16:59.450302 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:59.449990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerDied","Data":"55de90d1b8cb652143c29305c2bc826fef22c449ec7e56f551ca26e6b6ccf13f"}
Sep 18 16:16:59.450302 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:59.450040    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 18 16:16:59.749401 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:16:59.749316    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:17:00.212912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:00.212880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:17:00.212912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:00.212917    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:00.213117 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:00.212889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:00.213117 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:00.212990    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:17:00.213117 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:00.213057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:17:00.213225 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:00.213151    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:17:00.455065 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:00.454888    2563 generic.go:356] "Generic (PLEG): container finished" podID="8746475b-5bd4-4eb0-b54a-93ebca3ba19f" containerID="f495a076dc3ae627ca07728758ca0572d1b28ac9190d469fad3c7e0e45fd1644" exitCode=0
Sep 18 16:17:00.455422 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:00.454932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerDied","Data":"f495a076dc3ae627ca07728758ca0572d1b28ac9190d469fad3c7e0e45fd1644"}
Sep 18 16:17:00.465572 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:00.465517    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" podUID="64a4bc63-9c28-4830-81d2-8d2dd866b848" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 18 16:17:01.467406 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:01.467370    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5" podUID="64a4bc63-9c28-4830-81d2-8d2dd866b848" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 18 16:17:02.212598 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:02.212502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:02.212598 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:02.212510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:02.212826 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:02.212649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:17:02.212826 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:02.212653    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5ws42" podUID="dd636e67-a256-4028-b55d-7dbf12f724f7"
Sep 18 16:17:02.212826 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:02.212745    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bq84s" podUID="5bd67021-297e-41c1-b48a-ed9809e9cd2e"
Sep 18 16:17:02.212964 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:02.212836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:17:03.160300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.160267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:03.253034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.252980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:03.321285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.321252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:03.819173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.819082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:17:03.819318 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:03.819272    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:17:03.819404 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:03.819352    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:35.819332341 +0000 UTC m=+67.819593134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:17:03.866626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.866599    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeReady"
Sep 18 16:17:03.866791 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.866752    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:17:03.919848 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.919815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:03.920018 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:03.919927    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:17:03.920018 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:03.920011    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret podName:dd636e67-a256-4028-b55d-7dbf12f724f7 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:35.919991674 +0000 UTC m=+67.920252477 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret") pod "global-pull-secret-syncer-5ws42" (UID: "dd636e67-a256-4028-b55d-7dbf12f724f7") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:17:03.950695 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.950659    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tss2n"]
Sep 18 16:17:03.954010 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.953966    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9p64m"]
Sep 18 16:17:03.954162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.954100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:03.956266 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.956233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:03.959983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.959962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:17:03.960105 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.960055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccb8q\""
Sep 18 16:17:03.960191 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.960174    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:17:03.960754 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.960304    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qbbmj\""
Sep 18 16:17:03.960754 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.960357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:17:03.960754 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.960524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:17:03.960754 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.960638    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:17:03.974236 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.974177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9p64m"]
Sep 18 16:17:03.979478 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:03.979450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tss2n"]
Sep 18 16:17:04.020374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.020343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:04.020536 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.020490    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:17:04.020536 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.020507    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:17:04.020536 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.020517    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cf4mn for pod openshift-network-diagnostics/network-check-target-bq84s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:17:04.020714 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.020592    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn podName:5bd67021-297e-41c1-b48a-ed9809e9cd2e nodeName:}" failed. No retries permitted until 2026-09-18 16:17:36.020553659 +0000 UTC m=+68.020814452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cf4mn" (UniqueName: "kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn") pod "network-check-target-bq84s" (UID: "5bd67021-297e-41c1-b48a-ed9809e9cd2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:17:04.120917 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.120837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.120917 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.120873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqfg\" (UniqueName: \"kubernetes.io/projected/95792173-3a85-4a4e-bcb4-624522af1e2b-kube-api-access-ltqfg\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.120917 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.120897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5831d0a0-336e-41ba-9365-c73a416d6dd3-cert\") pod \"ingress-canary-9p64m\" (UID: \"5831d0a0-336e-41ba-9365-c73a416d6dd3\") " pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.121165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.120998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgx48\" (UniqueName: \"kubernetes.io/projected/5831d0a0-336e-41ba-9365-c73a416d6dd3-kube-api-access-mgx48\") pod \"ingress-canary-9p64m\" (UID: \"5831d0a0-336e-41ba-9365-c73a416d6dd3\") " pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.121165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.121043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95792173-3a85-4a4e-bcb4-624522af1e2b-config-volume\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.121165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.121082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95792173-3a85-4a4e-bcb4-624522af1e2b-tmp-dir\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.196342 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.196309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:04.212333 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.212306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:04.212497 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.212366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:17:04.212497 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.212373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:04.215666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.215276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:17:04.215666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.215391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ctnzv\""
Sep 18 16:17:04.215666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.215396    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:17:04.215666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.215504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-75jt8\""
Sep 18 16:17:04.215666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.215531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:17:04.215986 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.215685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:17:04.221700 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.221679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgx48\" (UniqueName: \"kubernetes.io/projected/5831d0a0-336e-41ba-9365-c73a416d6dd3-kube-api-access-mgx48\") pod \"ingress-canary-9p64m\" (UID: \"5831d0a0-336e-41ba-9365-c73a416d6dd3\") " pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.221827 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.221727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95792173-3a85-4a4e-bcb4-624522af1e2b-config-volume\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.221827 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.221771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95792173-3a85-4a4e-bcb4-624522af1e2b-tmp-dir\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.221827 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.221819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.221975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.221844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqfg\" (UniqueName: \"kubernetes.io/projected/95792173-3a85-4a4e-bcb4-624522af1e2b-kube-api-access-ltqfg\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.221975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.221869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5831d0a0-336e-41ba-9365-c73a416d6dd3-cert\") pod \"ingress-canary-9p64m\" (UID: \"5831d0a0-336e-41ba-9365-c73a416d6dd3\") " pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.222392 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.222165    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:04.222392 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.222232    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:04.722212574 +0000 UTC m=+36.722473385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:04.222392 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.222255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95792173-3a85-4a4e-bcb4-624522af1e2b-tmp-dir\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.222779 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.222757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95792173-3a85-4a4e-bcb4-624522af1e2b-config-volume\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.227073 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.227054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5831d0a0-336e-41ba-9365-c73a416d6dd3-cert\") pod \"ingress-canary-9p64m\" (UID: \"5831d0a0-336e-41ba-9365-c73a416d6dd3\") " pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.236951 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.236933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqfg\" (UniqueName: \"kubernetes.io/projected/95792173-3a85-4a4e-bcb4-624522af1e2b-kube-api-access-ltqfg\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.237252 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.237226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgx48\" (UniqueName: \"kubernetes.io/projected/5831d0a0-336e-41ba-9365-c73a416d6dd3-kube-api-access-mgx48\") pod \"ingress-canary-9p64m\" (UID: \"5831d0a0-336e-41ba-9365-c73a416d6dd3\") " pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.277607 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.277552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9p64m"
Sep 18 16:17:04.296262 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.296224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:04.726179 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:04.726150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:04.726354 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.726314    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:04.726410 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:04.726392    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:05.726369582 +0000 UTC m=+37.726630388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:05.099368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:05.099331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9p64m"]
Sep 18 16:17:05.103903 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:17:05.103872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5831d0a0_336e_41ba_9365_c73a416d6dd3.slice/crio-f8ffdd8533279947b570faafa87bd41b06e46c4b8d3ff549855522627bb9650e WatchSource:0}: Error finding container f8ffdd8533279947b570faafa87bd41b06e46c4b8d3ff549855522627bb9650e: Status 404 returned error can't find the container with id f8ffdd8533279947b570faafa87bd41b06e46c4b8d3ff549855522627bb9650e
Sep 18 16:17:05.467902 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:05.467865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9p64m" event={"ID":"5831d0a0-336e-41ba-9365-c73a416d6dd3","Type":"ContainerStarted","Data":"f8ffdd8533279947b570faafa87bd41b06e46c4b8d3ff549855522627bb9650e"}
Sep 18 16:17:05.470525 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:05.470501    2563 generic.go:356] "Generic (PLEG): container finished" podID="8746475b-5bd4-4eb0-b54a-93ebca3ba19f" containerID="a5e3569e3c6f8b6f886cfb76223a9d781913cb965febddafd822e97333d86ee0" exitCode=0
Sep 18 16:17:05.470619 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:05.470548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerDied","Data":"a5e3569e3c6f8b6f886cfb76223a9d781913cb965febddafd822e97333d86ee0"}
Sep 18 16:17:05.733091 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:05.733002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:05.733243 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:05.733168    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:05.733243 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:05.733238    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:07.733223244 +0000 UTC m=+39.733484034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:06.476153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:06.476120    2563 generic.go:356] "Generic (PLEG): container finished" podID="8746475b-5bd4-4eb0-b54a-93ebca3ba19f" containerID="ac53a5fa4c9b315002e9f1f8aeb487a9b9f3aee0fbc21d2ee89e4b73e8334154" exitCode=0
Sep 18 16:17:06.476543 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:06.476182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerDied","Data":"ac53a5fa4c9b315002e9f1f8aeb487a9b9f3aee0fbc21d2ee89e4b73e8334154"}
Sep 18 16:17:07.482078 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:07.482047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" event={"ID":"8746475b-5bd4-4eb0-b54a-93ebca3ba19f","Type":"ContainerStarted","Data":"e215c62739952e72def3d95f99330dd3b6cc4ba21dc13396b31aca9573974adf"}
Sep 18 16:17:07.509148 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:07.508882    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ndq2c" podStartSLOduration=7.706812712 podStartE2EDuration="37.50886135s" totalImagesPullingTime="29.802048638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:07.508323675 +0000 UTC m=+39.508584530" watchObservedRunningTime="2026-09-18 16:17:07.50886135 +0000 UTC m=+39.509122164"
Sep 18 16:17:07.741469 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:07.741430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.750444 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:07.750422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:07.750538 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:07.750526    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:07.750606 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:07.750597    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:11.750583775 +0000 UTC m=+43.750844563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:07.851211 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:07.851126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:07.972598 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:07.972549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:08.485447 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:08.485395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9p64m" event={"ID":"5831d0a0-336e-41ba-9365-c73a416d6dd3","Type":"ContainerStarted","Data":"0258e26b131dc63da31bd1641ee2061e48a8974690732078fcd5a96b1181a611"}
Sep 18 16:17:08.513212 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:08.513166    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9p64m" podStartSLOduration=3.152547765 podStartE2EDuration="5.513151603s" totalImagesPullingTime="2.360603838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:08.512013545 +0000 UTC m=+40.512274370" watchObservedRunningTime="2026-09-18 16:17:08.513151603 +0000 UTC m=+40.513412413"
Sep 18 16:17:08.948939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:08.948913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:09.049485 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:09.049454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:11.777708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:11.777661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:11.778163 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:11.777817    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:11.778163 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:11.777884    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:19.777867077 +0000 UTC m=+51.778127866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:19.830857 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:19.830815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:19.831425 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:19.830993    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:19.831425 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:19.831081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:35.831059722 +0000 UTC m=+67.831320534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:31.467388 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:31.467355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fw7m5"
Sep 18 16:17:31.622415 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:31.622387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:31.693641 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:31.693610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:17:35.837350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:35.837312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:17:35.837350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:35.837358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:17:35.837797 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:35.837473    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:35.837797 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:35.837538    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:18:07.837522389 +0000 UTC m=+99.837783201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:35.840035 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:35.840019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:17:35.848395 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:35.848377    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:17:35.848448 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:17:35.848436    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:39.848423425 +0000 UTC m=+131.848684213 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : secret "metrics-daemon-secret" not found
Sep 18 16:17:35.937715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:35.937679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:35.940375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:35.940357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:17:35.951834 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:35.951814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd636e67-a256-4028-b55d-7dbf12f724f7-original-pull-secret\") pod \"global-pull-secret-syncer-5ws42\" (UID: \"dd636e67-a256-4028-b55d-7dbf12f724f7\") " pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:36.023933 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.023897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5ws42"
Sep 18 16:17:36.038372 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.038348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:36.041136 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.041119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:17:36.050973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.050956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:17:36.070952 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.070928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4mn\" (UniqueName: \"kubernetes.io/projected/5bd67021-297e-41c1-b48a-ed9809e9cd2e-kube-api-access-cf4mn\") pod \"network-check-target-bq84s\" (UID: \"5bd67021-297e-41c1-b48a-ed9809e9cd2e\") " pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:36.329614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.329579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5ws42"]
Sep 18 16:17:36.333214 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:17:36.333190    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd636e67_a256_4028_b55d_7dbf12f724f7.slice/crio-dd0a327412f2eec49338c33318c468f021d9b64c2000eb447820671cba27479a WatchSource:0}: Error finding container dd0a327412f2eec49338c33318c468f021d9b64c2000eb447820671cba27479a: Status 404 returned error can't find the container with id dd0a327412f2eec49338c33318c468f021d9b64c2000eb447820671cba27479a
Sep 18 16:17:36.351805 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.351786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-75jt8\""
Sep 18 16:17:36.359983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.359930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:36.550632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.550598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ws42" event={"ID":"dd636e67-a256-4028-b55d-7dbf12f724f7","Type":"ContainerStarted","Data":"dd0a327412f2eec49338c33318c468f021d9b64c2000eb447820671cba27479a"}
Sep 18 16:17:36.642780 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:36.642750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bq84s"]
Sep 18 16:17:36.645790 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:17:36.645762    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd67021_297e_41c1_b48a_ed9809e9cd2e.slice/crio-3fc7d5ecc9d9d5381e3639eb8f924825732f3282d3adde59584c2a09bb193c9b WatchSource:0}: Error finding container 3fc7d5ecc9d9d5381e3639eb8f924825732f3282d3adde59584c2a09bb193c9b: Status 404 returned error can't find the container with id 3fc7d5ecc9d9d5381e3639eb8f924825732f3282d3adde59584c2a09bb193c9b
Sep 18 16:17:37.554919 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:37.554888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bq84s" event={"ID":"5bd67021-297e-41c1-b48a-ed9809e9cd2e","Type":"ContainerStarted","Data":"3fc7d5ecc9d9d5381e3639eb8f924825732f3282d3adde59584c2a09bb193c9b"}
Sep 18 16:17:41.567429 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:41.567392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5ws42" event={"ID":"dd636e67-a256-4028-b55d-7dbf12f724f7","Type":"ContainerStarted","Data":"159e35cd46fc005372c7c2fa68502b2c9188854b892dbc31853d669752677894"}
Sep 18 16:17:41.568909 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:41.568880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bq84s" event={"ID":"5bd67021-297e-41c1-b48a-ed9809e9cd2e","Type":"ContainerStarted","Data":"c90e0ba6f3bbcec84f3542f38d34b867def31083110d6ca47bacdb5b3deb18ec"}
Sep 18 16:17:41.569030 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:41.569014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:17:41.582190 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:41.582148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5ws42" podStartSLOduration=66.901779099 podStartE2EDuration="1m11.582133846s" totalImagesPullingTime="4.680354747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:41.581857131 +0000 UTC m=+73.582117962" watchObservedRunningTime="2026-09-18 16:17:41.582133846 +0000 UTC m=+73.582394656"
Sep 18 16:17:41.599877 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:17:41.599837    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bq84s" podStartSLOduration=67.236520653 podStartE2EDuration="1m11.599824079s" totalImagesPullingTime="4.363303426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:41.599198937 +0000 UTC m=+73.599459760" watchObservedRunningTime="2026-09-18 16:17:41.599824079 +0000 UTC m=+73.600084889"
Sep 18 16:18:07.852321 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:18:07.852223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:18:07.852730 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:18:07.852352    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:18:07.852730 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:18:07.852414    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:19:11.852396468 +0000 UTC m=+163.852657265 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:18:12.574503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:18:12.574470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bq84s"
Sep 18 16:18:39.857762 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:18:39.857701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:18:39.858287 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:18:39.857853    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:18:39.858287 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:18:39.857930    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs podName:66a797a2-9e7c-4c77-9523-2e62b0cccb79 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:41.857914513 +0000 UTC m=+253.858175307 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs") pod "network-metrics-daemon-7t76l" (UID: "66a797a2-9e7c-4c77-9523-2e62b0cccb79") : secret "metrics-daemon-secret" not found
Sep 18 16:19:06.971257 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:19:06.971200    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tss2n" podUID="95792173-3a85-4a4e-bcb4-624522af1e2b"
Sep 18 16:19:07.237058 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:19:07.236987    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7t76l" podUID="66a797a2-9e7c-4c77-9523-2e62b0cccb79"
Sep 18 16:19:07.772166 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:19:07.772090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tss2n"
Sep 18 16:19:11.866204 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:19:11.866096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:19:11.866617 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:19:11.866258    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:19:11.866617 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:19:11.866330    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls podName:95792173-3a85-4a4e-bcb4-624522af1e2b nodeName:}" failed. No retries permitted until 2026-09-18 16:21:13.866312979 +0000 UTC m=+285.866573773 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls") pod "dns-default-tss2n" (UID: "95792173-3a85-4a4e-bcb4-624522af1e2b") : secret "dns-default-metrics-tls" not found
Sep 18 16:19:20.213506 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:19:20.213466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:19:40.424275 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:19:40.424228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.295661 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.295626    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"]
Sep 18 16:20:13.297417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.297398    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.300042 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.300006    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 16:20:13.300316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.300178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.300316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.300291    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.300760 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.300735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ptpsm\""
Sep 18 16:20:13.300871 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.300785    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"]
Sep 18 16:20:13.301787 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.301770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 16:20:13.302339 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.302321    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"]
Sep 18 16:20:13.302467 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.302454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.303724 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.303708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"
Sep 18 16:20:13.304986 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.304965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9xpx5\""
Sep 18 16:20:13.305265 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.305249    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.306502 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.306485    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.306637 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.306518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.306637 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.306485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 16:20:13.306637 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.306543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.306791 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.306680    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l5vzt\""
Sep 18 16:20:13.323834 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.323813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"]
Sep 18 16:20:13.329844 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.329824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"]
Sep 18 16:20:13.343200 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.343177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"]
Sep 18 16:20:13.369169 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.369146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.369283 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.369175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bfl\" (UniqueName: \"kubernetes.io/projected/5564b299-dac8-44fa-ac85-4370d08c2bc8-kube-api-access-46bfl\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.369283 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.369208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5564b299-dac8-44fa-ac85-4370d08c2bc8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.369283 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.369226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhlk\" (UniqueName: \"kubernetes.io/projected/66a92c4b-715b-419d-bcc7-49933ba01916-kube-api-access-srhlk\") pod \"volume-data-source-validator-5c4db8bdf9-ldscf\" (UID: \"66a92c4b-715b-419d-bcc7-49933ba01916\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"
Sep 18 16:20:13.369283 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.369267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kctls\" (UniqueName: \"kubernetes.io/projected/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-kube-api-access-kctls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.369413 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.369288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.387384 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.387354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d877f4498-kz5rl"]
Sep 18 16:20:13.389000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.388987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.393675 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.393648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 16:20:13.393675 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.393647    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 16:20:13.393847 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.393755    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 16:20:13.393847 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.393822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ghgzf\""
Sep 18 16:20:13.394074 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.394050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 16:20:13.394190 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.394153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 16:20:13.394190 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.394157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 16:20:13.408549 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.408523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d877f4498-kz5rl"]
Sep 18 16:20:13.470474 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kctls\" (UniqueName: \"kubernetes.io/projected/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-kube-api-access-kctls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.470474 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-stats-auth\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68ph\" (UniqueName: \"kubernetes.io/projected/920459df-d1e0-476a-8a98-df1514196669-kube-api-access-p68ph\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bfl\" (UniqueName: \"kubernetes.io/projected/5564b299-dac8-44fa-ac85-4370d08c2bc8-kube-api-access-46bfl\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.470643    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5564b299-dac8-44fa-ac85-4370d08c2bc8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.470705    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls podName:6d01c33a-fcf2-4270-86c2-c7b2d233bf96 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.970690205 +0000 UTC m=+225.970950999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-sbnp4" (UID: "6d01c33a-fcf2-4270-86c2-c7b2d233bf96") : secret "samples-operator-tls" not found
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.470712    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:13.470720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhlk\" (UniqueName: \"kubernetes.io/projected/66a92c4b-715b-419d-bcc7-49933ba01916-kube-api-access-srhlk\") pod \"volume-data-source-validator-5c4db8bdf9-ldscf\" (UID: \"66a92c4b-715b-419d-bcc7-49933ba01916\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"
Sep 18 16:20:13.471102 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.471102 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.470766    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls podName:5564b299-dac8-44fa-ac85-4370d08c2bc8 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.970750979 +0000 UTC m=+225.971011768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bllvq" (UID: "5564b299-dac8-44fa-ac85-4370d08c2bc8") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:13.471102 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.470799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-default-certificate\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.471264 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.471246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5564b299-dac8-44fa-ac85-4370d08c2bc8-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.479409 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.479371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bfl\" (UniqueName: \"kubernetes.io/projected/5564b299-dac8-44fa-ac85-4370d08c2bc8-kube-api-access-46bfl\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.480192 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.480166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kctls\" (UniqueName: \"kubernetes.io/projected/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-kube-api-access-kctls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.480304 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.480288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhlk\" (UniqueName: \"kubernetes.io/projected/66a92c4b-715b-419d-bcc7-49933ba01916-kube-api-access-srhlk\") pod \"volume-data-source-validator-5c4db8bdf9-ldscf\" (UID: \"66a92c4b-715b-419d-bcc7-49933ba01916\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"
Sep 18 16:20:13.532744 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.532711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.572116 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.572057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.572116 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.572088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-default-certificate\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.572287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.572120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-stats-auth\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.572287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.572136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.572287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.572151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68ph\" (UniqueName: \"kubernetes.io/projected/920459df-d1e0-476a-8a98-df1514196669-kube-api-access-p68ph\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.572287 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.572235    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:20:13.572287 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.572237    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:14.072218799 +0000 UTC m=+226.072479605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:13.572539 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.572298    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:14.072279594 +0000 UTC m=+226.072540383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : secret "router-metrics-certs-default" not found
Sep 18 16:20:13.575147 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.575119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-default-certificate\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.575226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.575119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-stats-auth\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.582379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.582360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68ph\" (UniqueName: \"kubernetes.io/projected/920459df-d1e0-476a-8a98-df1514196669-kube-api-access-p68ph\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:13.617717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.617694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"
Sep 18 16:20:13.636003 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.635973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:13.891189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.891158    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf"]
Sep 18 16:20:13.895638 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:20:13.895609    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a92c4b_715b_419d_bcc7_49933ba01916.slice/crio-14b739a8198718b8490391e9241891b12b3d0db982088eb2d558c8ca807650cf WatchSource:0}: Error finding container 14b739a8198718b8490391e9241891b12b3d0db982088eb2d558c8ca807650cf: Status 404 returned error can't find the container with id 14b739a8198718b8490391e9241891b12b3d0db982088eb2d558c8ca807650cf
Sep 18 16:20:13.923233 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.923204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf" event={"ID":"66a92c4b-715b-419d-bcc7-49933ba01916","Type":"ContainerStarted","Data":"14b739a8198718b8490391e9241891b12b3d0db982088eb2d558c8ca807650cf"}
Sep 18 16:20:13.975301 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.975274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:13.975423 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:13.975311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:13.975423 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.975418    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:13.975507 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.975419    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:20:13.975507 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.975471    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls podName:5564b299-dac8-44fa-ac85-4370d08c2bc8 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:14.975457423 +0000 UTC m=+226.975718212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bllvq" (UID: "5564b299-dac8-44fa-ac85-4370d08c2bc8") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:13.975507 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:13.975484    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls podName:6d01c33a-fcf2-4270-86c2-c7b2d233bf96 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:14.975477715 +0000 UTC m=+226.975738504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-sbnp4" (UID: "6d01c33a-fcf2-4270-86c2-c7b2d233bf96") : secret "samples-operator-tls" not found
Sep 18 16:20:14.076550 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:14.076519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:14.076738 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:14.076585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:14.076738 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.076667    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:20:14.076738 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.076703    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:15.076682465 +0000 UTC m=+227.076943254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:14.076738 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.076734    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:15.076724613 +0000 UTC m=+227.076985402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : secret "router-metrics-certs-default" not found
Sep 18 16:20:14.927106 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:14.927076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf" event={"ID":"66a92c4b-715b-419d-bcc7-49933ba01916","Type":"ContainerStarted","Data":"15a0deda98d9af76c8ef445df71f0c27ab4d067ad1c5ed3421489c3dedd4f87a"}
Sep 18 16:20:14.943950 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:14.943895    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-ldscf" podStartSLOduration=1.019969222 podStartE2EDuration="1.943881227s" totalImagesPullingTime="923.912005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:14.94347412 +0000 UTC m=+226.943734943" watchObservedRunningTime="2026-09-18 16:20:14.943881227 +0000 UTC m=+226.944142038"
Sep 18 16:20:14.982887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:14.982859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:14.982999 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:14.982900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:14.983036 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.983018    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:20:14.983074 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.983062    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:14.983109 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.983093    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls podName:6d01c33a-fcf2-4270-86c2-c7b2d233bf96 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:16.983077089 +0000 UTC m=+228.983337878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-sbnp4" (UID: "6d01c33a-fcf2-4270-86c2-c7b2d233bf96") : secret "samples-operator-tls" not found
Sep 18 16:20:14.983147 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:14.983124    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls podName:5564b299-dac8-44fa-ac85-4370d08c2bc8 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:16.983110952 +0000 UTC m=+228.983371740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bllvq" (UID: "5564b299-dac8-44fa-ac85-4370d08c2bc8") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:15.043245 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:15.043220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:15.083811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:15.083749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:15.083811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:15.083799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:15.083917 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:15.083892    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:20:15.083950 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:15.083929    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:17.083912148 +0000 UTC m=+229.084172938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:15.083988 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:15.083950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:17.083943708 +0000 UTC m=+229.084204496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : secret "router-metrics-certs-default" not found
Sep 18 16:20:15.144234 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:15.144208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:16.997895 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:16.997846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:16.998288 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:16.997985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:16.998288 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:16.998003    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:16.998288 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:16.998062    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:20:16.998288 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:16.998074    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls podName:5564b299-dac8-44fa-ac85-4370d08c2bc8 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:20.998055899 +0000 UTC m=+232.998316714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bllvq" (UID: "5564b299-dac8-44fa-ac85-4370d08c2bc8") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:16.998288 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:16.998096    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls podName:6d01c33a-fcf2-4270-86c2-c7b2d233bf96 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:20.998086603 +0000 UTC m=+232.998347395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-sbnp4" (UID: "6d01c33a-fcf2-4270-86c2-c7b2d233bf96") : secret "samples-operator-tls" not found
Sep 18 16:20:17.098541 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.098507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:17.098657 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.098625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:17.098693 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:17.098653    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:20:17.098732 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:17.098721    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:21.098705862 +0000 UTC m=+233.098966650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : secret "router-metrics-certs-default" not found
Sep 18 16:20:17.098776 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:17.098742    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:21.098730352 +0000 UTC m=+233.098991141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:17.559665 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.559633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:17.615987 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.615948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:17.888006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.887981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:17.989160 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.989129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:17.995587 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.995553    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"]
Sep 18 16:20:17.997299 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.997286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"
Sep 18 16:20:17.999766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:17.999748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-x5qbk\""
Sep 18 16:20:18.000454 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.000440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 16:20:18.000512 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.000446    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 16:20:18.014403 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.014375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"]
Sep 18 16:20:18.088749 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.088719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:18.105912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.105889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4zb\" (UniqueName: \"kubernetes.io/projected/b0a56faa-83ce-4f22-a03d-8b50ac74f33d-kube-api-access-qt4zb\") pod \"migrator-76794cbd6c-4pp6f\" (UID: \"b0a56faa-83ce-4f22-a03d-8b50ac74f33d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"
Sep 18 16:20:18.188220 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.188133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:18.206530 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.206505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4zb\" (UniqueName: \"kubernetes.io/projected/b0a56faa-83ce-4f22-a03d-8b50ac74f33d-kube-api-access-qt4zb\") pod \"migrator-76794cbd6c-4pp6f\" (UID: \"b0a56faa-83ce-4f22-a03d-8b50ac74f33d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"
Sep 18 16:20:18.214704 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.214683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4zb\" (UniqueName: \"kubernetes.io/projected/b0a56faa-83ce-4f22-a03d-8b50ac74f33d-kube-api-access-qt4zb\") pod \"migrator-76794cbd6c-4pp6f\" (UID: \"b0a56faa-83ce-4f22-a03d-8b50ac74f33d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"
Sep 18 16:20:18.301048 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.301016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:18.306026 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.306013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"
Sep 18 16:20:18.546350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.546271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f5b8q"]
Sep 18 16:20:18.548277 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.548262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.551019 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.550994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:20:18.551161 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.551029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:20:18.551308 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.551288    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:20:18.551868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.551853    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:20:18.551930 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.551864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rrvrk\""
Sep 18 16:20:18.559111 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.559091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:18.562727 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.562704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5b8q"]
Sep 18 16:20:18.575988 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.575967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f"]
Sep 18 16:20:18.578825 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:20:18.578806    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a56faa_83ce_4f22_a03d_8b50ac74f33d.slice/crio-7417b5308520dd3541056ef6e5e230f98ae530d822d5fbfbabcac61ee65b9b90 WatchSource:0}: Error finding container 7417b5308520dd3541056ef6e5e230f98ae530d822d5fbfbabcac61ee65b9b90: Status 404 returned error can't find the container with id 7417b5308520dd3541056ef6e5e230f98ae530d822d5fbfbabcac61ee65b9b90
Sep 18 16:20:18.609989 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.609966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454a38f9-b610-48bf-8429-890bc8b5101c-crio-socket\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.610072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.610019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.610072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.610036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.610142 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.610096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454a38f9-b610-48bf-8429-890bc8b5101c-data-volume\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.610196 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.610182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rjm\" (UniqueName: \"kubernetes.io/projected/454a38f9-b610-48bf-8429-890bc8b5101c-kube-api-access-f8rjm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.710888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.710849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rjm\" (UniqueName: \"kubernetes.io/projected/454a38f9-b610-48bf-8429-890bc8b5101c-kube-api-access-f8rjm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.710888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.710887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454a38f9-b610-48bf-8429-890bc8b5101c-crio-socket\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.711066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.710926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.711066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.710942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.711066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.710962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454a38f9-b610-48bf-8429-890bc8b5101c-data-volume\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.711066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.710996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454a38f9-b610-48bf-8429-890bc8b5101c-crio-socket\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.711066 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:18.711052    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:18.711223 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:18.711074    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:18.711223 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:18.711129    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:19.211109949 +0000 UTC m=+231.211370756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:18.711223 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:18.711146    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:19.211137902 +0000 UTC m=+231.211398691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:18.711374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.711357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454a38f9-b610-48bf-8429-890bc8b5101c-data-volume\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.719670 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.719647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rjm\" (UniqueName: \"kubernetes.io/projected/454a38f9-b610-48bf-8429-890bc8b5101c-kube-api-access-f8rjm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:18.938620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:18.938583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f" event={"ID":"b0a56faa-83ce-4f22-a03d-8b50ac74f33d","Type":"ContainerStarted","Data":"7417b5308520dd3541056ef6e5e230f98ae530d822d5fbfbabcac61ee65b9b90"}
Sep 18 16:20:19.215696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:19.215599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:19.215696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:19.215637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:19.216121 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:19.215732    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:19.216121 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:19.215745    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:19.216121 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:19.215810    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:20.215789695 +0000 UTC m=+232.216050484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:19.216121 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:19.215831    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:20.215819286 +0000 UTC m=+232.216080082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:19.797331 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:19.797289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:19.862049 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:19.862008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:20.224938 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:20.224911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:20.224938 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:20.224941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:20.225338 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:20.225032    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:20.225338 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:20.225057    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:20.225338 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:20.225095    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:22.225077913 +0000 UTC m=+234.225338702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:20.225338 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:20.225110    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:22.225103932 +0000 UTC m=+234.225364721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:20.887676 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:20.887641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:20.945896 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:20.945856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f" event={"ID":"b0a56faa-83ce-4f22-a03d-8b50ac74f33d","Type":"ContainerStarted","Data":"b8748302299bdab88b117db82c56127bf4f4446813aae131cc00dd715e3abe44"}
Sep 18 16:20:20.946049 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:20.945904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f" event={"ID":"b0a56faa-83ce-4f22-a03d-8b50ac74f33d","Type":"ContainerStarted","Data":"f0da73a50184060e932ff92281842fb6a3e18afba8da1f187459b4b723a4fdd6"}
Sep 18 16:20:20.965852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:20.965805    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-4pp6f" podStartSLOduration=2.572986607 podStartE2EDuration="3.965790764s" totalImagesPullingTime="1.392804157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:20.965352772 +0000 UTC m=+232.965613582" watchObservedRunningTime="2026-09-18 16:20:20.965790764 +0000 UTC m=+232.966051575"
Sep 18 16:20:21.031075 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:21.031041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:21.031223 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:21.031085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:21.031223 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.031184    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:21.031223 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.031190    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 16:20:21.031401 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.031235    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls podName:5564b299-dac8-44fa-ac85-4370d08c2bc8 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:29.031221983 +0000 UTC m=+241.031482771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bllvq" (UID: "5564b299-dac8-44fa-ac85-4370d08c2bc8") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:21.031401 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.031249    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls podName:6d01c33a-fcf2-4270-86c2-c7b2d233bf96 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:29.03124291 +0000 UTC m=+241.031503699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-sbnp4" (UID: "6d01c33a-fcf2-4270-86c2-c7b2d233bf96") : secret "samples-operator-tls" not found
Sep 18 16:20:21.132044 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:21.132007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:21.132206 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:21.132061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:21.132206 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.132163    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:29.132145568 +0000 UTC m=+241.132406356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:21.132206 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.132174    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:20:21.132316 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:21.132228    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:29.132213243 +0000 UTC m=+241.132474032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : secret "router-metrics-certs-default" not found
Sep 18 16:20:21.305764 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:21.305691    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmmfb_29305122-d419-448f-9807-b92cc8861913/dns-node-resolver/0.log"
Sep 18 16:20:22.105257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:22.105230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxdtm_312dfd92-b94e-484d-821a-f9ffba57a32f/node-ca/0.log"
Sep 18 16:20:22.239532 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:22.239499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:22.239532 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:22.239530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:22.239751 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:22.239633    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:22.239751 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:22.239683    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:26.23967032 +0000 UTC m=+238.239931109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:22.240051 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:22.240031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454a38f9-b610-48bf-8429-890bc8b5101c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:22.390389 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:22.390357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:23.104979 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:23.104954    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9p64m_5831d0a0-336e-41ba-9365-c73a416d6dd3/serve-healthcheck-canary/0.log"
Sep 18 16:20:23.507174 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:23.507152    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4pp6f_b0a56faa-83ce-4f22-a03d-8b50ac74f33d/migrator/0.log"
Sep 18 16:20:23.707746 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:23.707716    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4pp6f_b0a56faa-83ce-4f22-a03d-8b50ac74f33d/graceful-termination/0.log"
Sep 18 16:20:23.876140 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:23.876067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:23.993004 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:23.992974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:24.122706 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:24.122674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:24.332972 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:24.332936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:24.748216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:24.748183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:25.358687 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:25.358652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:25.973866 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:25.973835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:26.133084 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:26.133053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:26.274380 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:26.274299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:26.274518 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:26.274439    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:26.274518 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:26.274504    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls podName:454a38f9-b610-48bf-8429-890bc8b5101c nodeName:}" failed. No retries permitted until 2026-09-18 16:20:34.274486443 +0000 UTC m=+246.274747231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f5b8q" (UID: "454a38f9-b610-48bf-8429-890bc8b5101c") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:26.333880 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:26.333851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:26.540755 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:26.540689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:27.151507 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:27.151464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:27.334909 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:27.334880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:27.732886 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:27.732854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:27.829722 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:27.829694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:28.138119 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:28.138094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:28.539631 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:28.539528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:29.097656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.097619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:29.097656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.097660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:29.097947 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:29.097760    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:29.097947 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:29.097826    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls podName:5564b299-dac8-44fa-ac85-4370d08c2bc8 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:45.097809518 +0000 UTC m=+257.098070323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-bllvq" (UID: "5564b299-dac8-44fa-ac85-4370d08c2bc8") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:20:29.100225 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.100204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d01c33a-fcf2-4270-86c2-c7b2d233bf96-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-sbnp4\" (UID: \"6d01c33a-fcf2-4270-86c2-c7b2d233bf96\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:29.198529 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.198479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:29.198731 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.198549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:29.198731 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:20:29.198662    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle podName:920459df-d1e0-476a-8a98-df1514196669 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:45.198645656 +0000 UTC m=+257.198906445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle") pod "router-default-6d877f4498-kz5rl" (UID: "920459df-d1e0-476a-8a98-df1514196669") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:29.201016 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.200995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/920459df-d1e0-476a-8a98-df1514196669-metrics-certs\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:29.213170 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.213153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"
Sep 18 16:20:29.479171 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.479131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4"]
Sep 18 16:20:29.899463 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.899432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:29.974334 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.974303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4" event={"ID":"6d01c33a-fcf2-4270-86c2-c7b2d233bf96","Type":"ContainerStarted","Data":"abcef6871a8e6426125adc8ee70976e9b69d655aa9ddde81ab2d27745675f099"}
Sep 18 16:20:29.997621 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:29.997588    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:30.138904 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:30.138861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:30.337324 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:30.337244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:30.942499 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:30.942465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.552462 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:31.552436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.730874 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:31.730843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.934102 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:31.934069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:31.981819 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:31.981790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4" event={"ID":"6d01c33a-fcf2-4270-86c2-c7b2d233bf96","Type":"ContainerStarted","Data":"9ed1f5067f04498f6e7c5563aea6c8d5f3d76d2d2e8b1bd878cf244781e09abe"}
Sep 18 16:20:31.982189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:31.981838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4" event={"ID":"6d01c33a-fcf2-4270-86c2-c7b2d233bf96","Type":"ContainerStarted","Data":"d58bd81280532283a8346a060fb40340c4d042eb6b04ed06c89d7d934666c672"}
Sep 18 16:20:32.009595 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:32.009532    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-sbnp4" podStartSLOduration=17.504282326 podStartE2EDuration="19.009517689s" totalImagesPullingTime="1.505235363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:32.00795981 +0000 UTC m=+244.008220620" watchObservedRunningTime="2026-09-18 16:20:32.009517689 +0000 UTC m=+244.009778501"
Sep 18 16:20:32.335806 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:32.335723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:32.750307 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:32.750278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:32.824985 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:32.824949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:32.935644 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:32.935611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:33.429482 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:33.429450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:33.531176 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:33.531148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:34.130236 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.130205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:34.337038 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.337009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:34.337271 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.337251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:34.339645 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.339618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454a38f9-b610-48bf-8429-890bc8b5101c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f5b8q\" (UID: \"454a38f9-b610-48bf-8429-890bc8b5101c\") " pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:34.460426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.460399    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rrvrk\""
Sep 18 16:20:34.468009 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.467994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f5b8q"
Sep 18 16:20:34.477519 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.477462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:34.546135 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.546106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:34.742784 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.742713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f5b8q"]
Sep 18 16:20:34.746457 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:20:34.746432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454a38f9_b610_48bf_8429_890bc8b5101c.slice/crio-e6faf457580f6654991ebff17118c6e4c5308be8ebf406a0bd1149e24633be1e WatchSource:0}: Error finding container e6faf457580f6654991ebff17118c6e4c5308be8ebf406a0bd1149e24633be1e: Status 404 returned error can't find the container with id e6faf457580f6654991ebff17118c6e4c5308be8ebf406a0bd1149e24633be1e
Sep 18 16:20:34.991543 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.991504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5b8q" event={"ID":"454a38f9-b610-48bf-8429-890bc8b5101c","Type":"ContainerStarted","Data":"366cd12bf932d91f8da59bdc60079a3a6c0d7c684e1d8502316abc689b0dbbcc"}
Sep 18 16:20:34.991724 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:34.991576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5b8q" event={"ID":"454a38f9-b610-48bf-8429-890bc8b5101c","Type":"ContainerStarted","Data":"e6faf457580f6654991ebff17118c6e4c5308be8ebf406a0bd1149e24633be1e"}
Sep 18 16:20:35.996178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:35.996144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5b8q" event={"ID":"454a38f9-b610-48bf-8429-890bc8b5101c","Type":"ContainerStarted","Data":"a5450b70fc3f5fd150c2d43f12769bcb5ba92a3d3ed827db240df300791b3d50"}
Sep 18 16:20:36.735786 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:36.735754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.837260 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:36.837232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:36.904689 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:36.904638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:37.049569 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:37.049524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:37.421549 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:37.421517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:37.488081 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:37.488051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:37.809373 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:37.809291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:37.883508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:37.883474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:38.003273 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.003234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f5b8q" event={"ID":"454a38f9-b610-48bf-8429-890bc8b5101c","Type":"ContainerStarted","Data":"4dd3f317f50a3aa9df46d10fbc27c1ea3807c076aa7952ede634189c1bc8b569"}
Sep 18 16:20:38.023378 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.023330    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f5b8q" podStartSLOduration=17.916842279 podStartE2EDuration="20.023316622s" totalImagesPullingTime="2.106474343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:20:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:38.02211254 +0000 UTC m=+250.022373362" watchObservedRunningTime="2026-09-18 16:20:38.023316622 +0000 UTC m=+250.023577429"
Sep 18 16:20:38.046288 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.046262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:38.117324 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.117251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:38.451161 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.451129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:38.625464 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.625433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:38.725346 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:38.725247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:39.481269 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:39.481237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:39.566850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:39.566822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:40.698047 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:40.698018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:41.896879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:41.896847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:20:41.899219 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:41.899202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a797a2-9e7c-4c77-9523-2e62b0cccb79-metrics-certs\") pod \"network-metrics-daemon-7t76l\" (UID: \"66a797a2-9e7c-4c77-9523-2e62b0cccb79\") " pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:20:42.117451 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:42.117420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ctnzv\""
Sep 18 16:20:42.125173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:42.125154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7t76l"
Sep 18 16:20:42.399437 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:42.399403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7t76l"]
Sep 18 16:20:42.403424 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:20:42.403401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a797a2_9e7c_4c77_9523_2e62b0cccb79.slice/crio-a1de8ede6c2ae5ee4155d0aed1cdb041147fe71dc520f270c6208b03b960243d WatchSource:0}: Error finding container a1de8ede6c2ae5ee4155d0aed1cdb041147fe71dc520f270c6208b03b960243d: Status 404 returned error can't find the container with id a1de8ede6c2ae5ee4155d0aed1cdb041147fe71dc520f270c6208b03b960243d
Sep 18 16:20:43.021026 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:43.020990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7t76l" event={"ID":"66a797a2-9e7c-4c77-9523-2e62b0cccb79","Type":"ContainerStarted","Data":"a1de8ede6c2ae5ee4155d0aed1cdb041147fe71dc520f270c6208b03b960243d"}
Sep 18 16:20:44.025873 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:44.025838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7t76l" event={"ID":"66a797a2-9e7c-4c77-9523-2e62b0cccb79","Type":"ContainerStarted","Data":"ae846a851920afbf2e62109fa2e918d2f9e8fbc9de4cbbddb687d29103235ac3"}
Sep 18 16:20:44.026248 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:44.025914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7t76l" event={"ID":"66a797a2-9e7c-4c77-9523-2e62b0cccb79","Type":"ContainerStarted","Data":"3e897d0c3c5271367e20c249a696dd7471835454642f5ef2b8c39df11b9d2461"}
Sep 18 16:20:44.048518 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:44.048471    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7t76l" podStartSLOduration=253.19527288 podStartE2EDuration="4m14.048456853s" totalImagesPullingTime="853.183973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:44.047196751 +0000 UTC m=+256.047457579" watchObservedRunningTime="2026-09-18 16:20:44.048456853 +0000 UTC m=+256.048717667"
Sep 18 16:20:44.998764 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:44.998735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:45.074850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.074818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:45.119703 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.119672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:45.122260 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.122242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5564b299-dac8-44fa-ac85-4370d08c2bc8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-bllvq\" (UID: \"5564b299-dac8-44fa-ac85-4370d08c2bc8\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:45.220362 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.220325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:45.220950 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.220930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/920459df-d1e0-476a-8a98-df1514196669-service-ca-bundle\") pod \"router-default-6d877f4498-kz5rl\" (UID: \"920459df-d1e0-476a-8a98-df1514196669\") " pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:45.410488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.410455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ptpsm\""
Sep 18 16:20:45.418546 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.418526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"
Sep 18 16:20:45.500266 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.500230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ghgzf\""
Sep 18 16:20:45.508288 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.508272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:45.687859 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.687785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq"]
Sep 18 16:20:45.690913 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:20:45.690889    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5564b299_dac8_44fa_ac85_4370d08c2bc8.slice/crio-751a9251aad50cd4b783674a080b01d2cbbb1d574eb1a3d6aec693cdee4ef947 WatchSource:0}: Error finding container 751a9251aad50cd4b783674a080b01d2cbbb1d574eb1a3d6aec693cdee4ef947: Status 404 returned error can't find the container with id 751a9251aad50cd4b783674a080b01d2cbbb1d574eb1a3d6aec693cdee4ef947
Sep 18 16:20:45.774932 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:45.774897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d877f4498-kz5rl"]
Sep 18 16:20:45.778429 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:20:45.778408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920459df_d1e0_476a_8a98_df1514196669.slice/crio-9ded393cdc21f6c0089f1c22619f8c934e761de533e1ced139cd9a121b8db001 WatchSource:0}: Error finding container 9ded393cdc21f6c0089f1c22619f8c934e761de533e1ced139cd9a121b8db001: Status 404 returned error can't find the container with id 9ded393cdc21f6c0089f1c22619f8c934e761de533e1ced139cd9a121b8db001
Sep 18 16:20:46.033501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:46.033409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d877f4498-kz5rl" event={"ID":"920459df-d1e0-476a-8a98-df1514196669","Type":"ContainerStarted","Data":"be8365bb8cbf480b59d2e5c6c30debebfd254bd78d64d8a07b3eaa6fc09551c2"}
Sep 18 16:20:46.033501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:46.033468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d877f4498-kz5rl" event={"ID":"920459df-d1e0-476a-8a98-df1514196669","Type":"ContainerStarted","Data":"9ded393cdc21f6c0089f1c22619f8c934e761de533e1ced139cd9a121b8db001"}
Sep 18 16:20:46.034613 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:46.034591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq" event={"ID":"5564b299-dac8-44fa-ac85-4370d08c2bc8","Type":"ContainerStarted","Data":"751a9251aad50cd4b783674a080b01d2cbbb1d574eb1a3d6aec693cdee4ef947"}
Sep 18 16:20:46.054510 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:46.054464    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6d877f4498-kz5rl" podStartSLOduration=33.054449202 podStartE2EDuration="33.054449202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:46.054014653 +0000 UTC m=+258.054275485" watchObservedRunningTime="2026-09-18 16:20:46.054449202 +0000 UTC m=+258.054710013"
Sep 18 16:20:46.509186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:46.509147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:46.511707 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:46.511685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:47.038633 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:47.038596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:47.039919 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:47.039898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d877f4498-kz5rl"
Sep 18 16:20:47.772267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:47.772233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:47.837639 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:47.837609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:48.042904 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:48.042870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq" event={"ID":"5564b299-dac8-44fa-ac85-4370d08c2bc8","Type":"ContainerStarted","Data":"724ba338e6f141ff99817dd9af59670abfdb4cef0c7ec46e411e37f1e4fb1ec9"}
Sep 18 16:20:48.050852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:48.050828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:48.062963 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:48.062921    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-bllvq" podStartSLOduration=33.129700965 podStartE2EDuration="35.062906125s" totalImagesPullingTime="1.93320516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:48.061845239 +0000 UTC m=+260.062106061" watchObservedRunningTime="2026-09-18 16:20:48.062906125 +0000 UTC m=+260.063166935"
Sep 18 16:20:48.148429 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:48.148401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:48.247194 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:48.247163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:48.619097 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:48.619058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:49.921073 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:49.921037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:20:49.984844 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:20:49.984813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:01.471346 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:01.471316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:08.226363 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:08.226319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:08.283614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:08.283575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:10.773601 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:21:10.773518    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tss2n" podUID="95792173-3a85-4a4e-bcb4-624522af1e2b"
Sep 18 16:21:11.114832 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:11.114752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tss2n"
Sep 18 16:21:13.917840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:13.917805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:21:13.920249 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:13.920225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95792173-3a85-4a4e-bcb4-624522af1e2b-metrics-tls\") pod \"dns-default-tss2n\" (UID: \"95792173-3a85-4a4e-bcb4-624522af1e2b\") " pod="openshift-dns/dns-default-tss2n"
Sep 18 16:21:14.118362 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:14.118331    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ccb8q\""
Sep 18 16:21:14.126633 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:14.126610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tss2n"
Sep 18 16:21:14.395952 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:14.395919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tss2n"]
Sep 18 16:21:14.399716 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:21:14.399687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95792173_3a85_4a4e_bcb4_624522af1e2b.slice/crio-8aa13ee794d777a66f0d528daab463b0c67a79c246eb1848eeb352fa7460532d WatchSource:0}: Error finding container 8aa13ee794d777a66f0d528daab463b0c67a79c246eb1848eeb352fa7460532d: Status 404 returned error can't find the container with id 8aa13ee794d777a66f0d528daab463b0c67a79c246eb1848eeb352fa7460532d
Sep 18 16:21:15.128499 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:15.128460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tss2n" event={"ID":"95792173-3a85-4a4e-bcb4-624522af1e2b","Type":"ContainerStarted","Data":"8aa13ee794d777a66f0d528daab463b0c67a79c246eb1848eeb352fa7460532d"}
Sep 18 16:21:17.136666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:17.136628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tss2n" event={"ID":"95792173-3a85-4a4e-bcb4-624522af1e2b","Type":"ContainerStarted","Data":"e8c7ce2ac94bc0f299af37221836d2db3f85779b79828efd936b37750089cc3c"}
Sep 18 16:21:17.137071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:17.136692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tss2n" event={"ID":"95792173-3a85-4a4e-bcb4-624522af1e2b","Type":"ContainerStarted","Data":"20b8b39e11c43a4b6d1b66dbe461aa49b4290822c6e796bea41b206faf5cff30"}
Sep 18 16:21:17.137071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:17.136722    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tss2n"
Sep 18 16:21:17.155236 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:17.155188    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tss2n" podStartSLOduration=252.478679314 podStartE2EDuration="4m14.15517548s" totalImagesPullingTime="1.676496166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:17.153967397 +0000 UTC m=+289.154228233" watchObservedRunningTime="2026-09-18 16:21:17.15517548 +0000 UTC m=+289.155436291"
Sep 18 16:21:17.817454 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:17.817419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:17.892554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:17.892529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:27.142502 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:27.142474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tss2n"
Sep 18 16:21:27.340960 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:27.340922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:27.436696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:27.436626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:30.074238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:30.074207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:21:30.077707 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:30.077684    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:21:30.081386 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:30.081368    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:21:36.928746 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:36.928713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:37.029111 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:37.029083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:46.489999 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:46.489959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:46.590140 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:46.590110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:53.378704 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:53.378674    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tss2n_95792173-3a85-4a4e-bcb4-624522af1e2b/dns/0.log"
Sep 18 16:21:53.575305 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:53.575275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tss2n_95792173-3a85-4a4e-bcb4-624522af1e2b/kube-rbac-proxy/0.log"
Sep 18 16:21:53.974814 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:53.974790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmmfb_29305122-d419-448f-9807-b92cc8861913/dns-node-resolver/0.log"
Sep 18 16:21:55.175124 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:55.175096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxdtm_312dfd92-b94e-484d-821a-f9ffba57a32f/node-ca/0.log"
Sep 18 16:21:55.777326 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:55.777297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d877f4498-kz5rl_920459df-d1e0-476a-8a98-df1514196669/router/0.log"
Sep 18 16:21:56.013369 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:56.013341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:21:56.174603 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:56.174553    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9p64m_5831d0a0-336e-41ba-9365-c73a416d6dd3/serve-healthcheck-canary/0.log"
Sep 18 16:21:56.576095 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:56.576014    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4pp6f_b0a56faa-83ce-4f22-a03d-8b50ac74f33d/migrator/0.log"
Sep 18 16:21:56.775624 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:21:56.775597    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4pp6f_b0a56faa-83ce-4f22-a03d-8b50ac74f33d/graceful-termination/0.log"
Sep 18 16:22:03.214068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:03.214031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:05.490915 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:05.490882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:05.556188 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:05.556157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.078624 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:15.078512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:15.144067 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:15.144018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:21.395464 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:21.395433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:21.534449 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:21.534371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:21.643224 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:21.643201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:24.637805 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:24.637769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:25.253554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:25.253519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:25.335792 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:25.335752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:25.845902 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:25.845869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:28.318193 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:28.318161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:28.719182 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:28.719154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:28.792717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:28.792691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:30.442639 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.442608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:30.652543 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.652511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:30.737518 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.735099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7plgk"]
Sep 18 16:22:30.738537 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.738512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.741859 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.741834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:22:30.741985 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.741910    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4wvnt\""
Sep 18 16:22:30.742155 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.742143    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:22:30.742262 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.742248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:22:30.744201 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.744183    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:22:30.767263 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.767188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:30.791233 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a102568-21a5-497b-8a2e-cb67a7f3106f-metrics-client-ca\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791233 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-textfile\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-root\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-wtmp\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7pk\" (UniqueName: \"kubernetes.io/projected/1a102568-21a5-497b-8a2e-cb67a7f3106f-kube-api-access-zq7pk\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-sys\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.791427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.791429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-tls\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.872284 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.872247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:30.892349 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7pk\" (UniqueName: \"kubernetes.io/projected/1a102568-21a5-497b-8a2e-cb67a7f3106f-kube-api-access-zq7pk\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-sys\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-tls\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a102568-21a5-497b-8a2e-cb67a7f3106f-metrics-client-ca\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-sys\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-textfile\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-root\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-wtmp\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:22:30.892592    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:22:30.892685    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-tls podName:1a102568-21a5-497b-8a2e-cb67a7f3106f nodeName:}" failed. No retries permitted until 2026-09-18 16:22:31.392663629 +0000 UTC m=+363.392924419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-tls") pod "node-exporter-7plgk" (UID: "1a102568-21a5-497b-8a2e-cb67a7f3106f") : secret "node-exporter-tls" not found
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892723    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-wtmp\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.892775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a102568-21a5-497b-8a2e-cb67a7f3106f-root\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.893076 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-textfile\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.893076 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.892986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.893076 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.893051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a102568-21a5-497b-8a2e-cb67a7f3106f-metrics-client-ca\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.894931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.894913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.919378 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.919355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7pk\" (UniqueName: \"kubernetes.io/projected/1a102568-21a5-497b-8a2e-cb67a7f3106f-kube-api-access-zq7pk\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:30.978703 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:30.978667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.079372 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.079290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.179768 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.179737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.397197 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.397165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-tls\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:31.399594 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.399554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a102568-21a5-497b-8a2e-cb67a7f3106f-node-exporter-tls\") pod \"node-exporter-7plgk\" (UID: \"1a102568-21a5-497b-8a2e-cb67a7f3106f\") " pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:31.455795 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.455768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.647621 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.647531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7plgk"
Sep 18 16:22:31.654674 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.654647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.656890 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:22:31.656863    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a102568_21a5_497b_8a2e_cb67a7f3106f.slice/crio-e505d1ad3edc3e1527336ea904eacfa652df5fd07accd6cd2ff46c176377b27b WatchSource:0}: Error finding container e505d1ad3edc3e1527336ea904eacfa652df5fd07accd6cd2ff46c176377b27b: Status 404 returned error can't find the container with id e505d1ad3edc3e1527336ea904eacfa652df5fd07accd6cd2ff46c176377b27b
Sep 18 16:22:31.658497 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.658483    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:22:31.760202 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.760163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.781086 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.781057    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:22:31.784201 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.784183    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.787499 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.787477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 16:22:31.787677 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.787656    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 16:22:31.787797 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.787745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 16:22:31.787797 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.787745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 16:22:31.787914 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.787861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 16:22:31.788031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.788012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 16:22:31.788092 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.788051    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 16:22:31.788092 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.788055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 16:22:31.789729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.789714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 16:22:31.790233 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.790214    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vldj4\""
Sep 18 16:22:31.800174 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-web-config\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-out\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqs2p\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-kube-api-access-mqs2p\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.800509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.800507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.801642 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.801623    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:22:31.854574 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.854526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:31.901894 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.901830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.901894 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.901859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.901894 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.901881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902077 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:22:31.901988    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle podName:aef632de-d4fa-4614-94c3-5f6c664b21d6 nodeName:}" failed. No retries permitted until 2026-09-18 16:22:32.40196635 +0000 UTC m=+364.402227142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6") : configmap references non-existent config key: ca-bundle.crt
Sep 18 16:22:31.902077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqs2p\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-kube-api-access-mqs2p\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-web-config\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902267 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.902595 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.902333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-out\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.903643 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.903250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.903643 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.903599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.905190 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.905165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.906202 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.906128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.906202 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.906174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.906356 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.906283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-web-config\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.906609 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.906591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-out\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.906675 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.906642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.906966 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.906949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.907161 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.907141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.907532 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.907513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:31.913207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:31.913185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqs2p\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-kube-api-access-mqs2p\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:32.379839 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:32.379740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7plgk" event={"ID":"1a102568-21a5-497b-8a2e-cb67a7f3106f","Type":"ContainerStarted","Data":"e505d1ad3edc3e1527336ea904eacfa652df5fd07accd6cd2ff46c176377b27b"}
Sep 18 16:22:32.407114 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:32.407073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:32.408018 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:32.408000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:32.433747 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:32.433705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:32.694014 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:32.693975    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:22:32.983365 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:32.983278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:22:32.988024 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:22:32.988002    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef632de_d4fa_4614_94c3_5f6c664b21d6.slice/crio-7b98bb68cc6e532962cf1a6aa830daa9410889d0364600fcc73090288faa62a9 WatchSource:0}: Error finding container 7b98bb68cc6e532962cf1a6aa830daa9410889d0364600fcc73090288faa62a9: Status 404 returned error can't find the container with id 7b98bb68cc6e532962cf1a6aa830daa9410889d0364600fcc73090288faa62a9
Sep 18 16:22:33.383894 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:33.383813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"7b98bb68cc6e532962cf1a6aa830daa9410889d0364600fcc73090288faa62a9"}
Sep 18 16:22:33.385365 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:33.385332    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a102568-21a5-497b-8a2e-cb67a7f3106f" containerID="65ebaa3d13dd0b7597ac08e311099a072d42ac7306fd51977cbe61de97f77fd7" exitCode=0
Sep 18 16:22:33.385480 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:33.385385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7plgk" event={"ID":"1a102568-21a5-497b-8a2e-cb67a7f3106f","Type":"ContainerDied","Data":"65ebaa3d13dd0b7597ac08e311099a072d42ac7306fd51977cbe61de97f77fd7"}
Sep 18 16:22:33.653111 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:33.653073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:33.754961 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:33.754915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:33.834128 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:33.834083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:34.025111 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.025030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:34.361440 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.361354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:34.390620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.390591    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62" exitCode=0
Sep 18 16:22:34.390780 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.390683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"}
Sep 18 16:22:34.393037 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.393015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7plgk" event={"ID":"1a102568-21a5-497b-8a2e-cb67a7f3106f","Type":"ContainerStarted","Data":"32ee25cfe05561f735774865bd7232f751f246500622fe29b000c619df29db30"}
Sep 18 16:22:34.393126 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.393064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7plgk" event={"ID":"1a102568-21a5-497b-8a2e-cb67a7f3106f","Type":"ContainerStarted","Data":"ba83db87d44d673e830f39e33db6f6f45702c80c881dd740564c1d0956239d28"}
Sep 18 16:22:34.437931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.437898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:34.442218 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.442179    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7plgk" podStartSLOduration=3.331197153 podStartE2EDuration="4.442166129s" totalImagesPullingTime="1.043353609s" totalInitContainerRuntime="67.615367ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:34.441777968 +0000 UTC m=+366.442038800" watchObservedRunningTime="2026-09-18 16:22:34.442166129 +0000 UTC m=+366.442426992"
Sep 18 16:22:34.502162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:34.502112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:35.051956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.051921    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d586cdf54-5f8fn"]
Sep 18 16:22:35.055163 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.055133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.069075 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.069052    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rlwrw\""
Sep 18 16:22:35.069202 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.069111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-13vva5s3anj48\""
Sep 18 16:22:35.069202 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.069168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:22:35.069329 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.069168    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 16:22:35.069446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.069432    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 16:22:35.077402 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.077380    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 16:22:35.082353 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.082310    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d586cdf54-5f8fn"]
Sep 18 16:22:35.128676 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50ddce52-3943-4e3d-ac23-24bf7f669fd8-metrics-server-audit-profiles\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.128818 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-secret-metrics-server-tls\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.128818 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50ddce52-3943-4e3d-ac23-24bf7f669fd8-audit-log\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.128943 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-client-ca-bundle\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.128943 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ddce52-3943-4e3d-ac23-24bf7f669fd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.128943 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-secret-metrics-server-client-certs\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.129072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.128959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knvh\" (UniqueName: \"kubernetes.io/projected/50ddce52-3943-4e3d-ac23-24bf7f669fd8-kube-api-access-7knvh\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.229663 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50ddce52-3943-4e3d-ac23-24bf7f669fd8-metrics-server-audit-profiles\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.229855 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-secret-metrics-server-tls\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.229855 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50ddce52-3943-4e3d-ac23-24bf7f669fd8-audit-log\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.229855 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-client-ca-bundle\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.230017 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ddce52-3943-4e3d-ac23-24bf7f669fd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.230017 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-secret-metrics-server-client-certs\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.230017 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.229914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knvh\" (UniqueName: \"kubernetes.io/projected/50ddce52-3943-4e3d-ac23-24bf7f669fd8-kube-api-access-7knvh\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.230224 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.230199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50ddce52-3943-4e3d-ac23-24bf7f669fd8-audit-log\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.230519 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.230487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50ddce52-3943-4e3d-ac23-24bf7f669fd8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.230926 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.230903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50ddce52-3943-4e3d-ac23-24bf7f669fd8-metrics-server-audit-profiles\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.232302 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.232277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-secret-metrics-server-tls\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.232418 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.232342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-secret-metrics-server-client-certs\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.232418 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.232401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50ddce52-3943-4e3d-ac23-24bf7f669fd8-client-ca-bundle\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.239527 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.239507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knvh\" (UniqueName: \"kubernetes.io/projected/50ddce52-3943-4e3d-ac23-24bf7f669fd8-kube-api-access-7knvh\") pod \"metrics-server-d586cdf54-5f8fn\" (UID: \"50ddce52-3943-4e3d-ac23-24bf7f669fd8\") " pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.366120 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.366036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:35.480207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.480169    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"]
Sep 18 16:22:35.483417 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.483393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:35.484458 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.484438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:35.487477 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.487460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6hnjt\""
Sep 18 16:22:35.488234 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.488215    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 16:22:35.496893 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.496872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"]
Sep 18 16:22:35.531640 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.531589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9170ef7-39a5-4934-8499-b2fc1fe927d0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:35.531785 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.531660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtzx2\" (UniqueName: \"kubernetes.io/projected/b9170ef7-39a5-4934-8499-b2fc1fe927d0-kube-api-access-jtzx2\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:35.633073 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.633005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9170ef7-39a5-4934-8499-b2fc1fe927d0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:35.633073 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.633047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtzx2\" (UniqueName: \"kubernetes.io/projected/b9170ef7-39a5-4934-8499-b2fc1fe927d0-kube-api-access-jtzx2\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:35.633286 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:22:35.633173    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 16:22:35.633286 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:22:35.633241    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9170ef7-39a5-4934-8499-b2fc1fe927d0-monitoring-plugin-cert podName:b9170ef7-39a5-4934-8499-b2fc1fe927d0 nodeName:}" failed. No retries permitted until 2026-09-18 16:22:36.133222213 +0000 UTC m=+368.133483006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b9170ef7-39a5-4934-8499-b2fc1fe927d0-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-zv6kw" (UID: "b9170ef7-39a5-4934-8499-b2fc1fe927d0") : secret "monitoring-plugin-cert" not found
Sep 18 16:22:35.642653 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.642616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtzx2\" (UniqueName: \"kubernetes.io/projected/b9170ef7-39a5-4934-8499-b2fc1fe927d0-kube-api-access-jtzx2\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:35.648509 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.648484    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d586cdf54-5f8fn"]
Sep 18 16:22:35.655901 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.655866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:35.725989 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:35.725952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:35.886291 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:22:35.886214    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ddce52_3943_4e3d_ac23_24bf7f669fd8.slice/crio-4b974ede0aec70f1c3ac81de4a01384fbd3a6558ce1cabd8cbdcf8359baf6820 WatchSource:0}: Error finding container 4b974ede0aec70f1c3ac81de4a01384fbd3a6558ce1cabd8cbdcf8359baf6820: Status 404 returned error can't find the container with id 4b974ede0aec70f1c3ac81de4a01384fbd3a6558ce1cabd8cbdcf8359baf6820
Sep 18 16:22:36.136851 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.136795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9170ef7-39a5-4934-8499-b2fc1fe927d0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:36.139081 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.139061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9170ef7-39a5-4934-8499-b2fc1fe927d0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-zv6kw\" (UID: \"b9170ef7-39a5-4934-8499-b2fc1fe927d0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:36.395196 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.395164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:36.405873 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.405847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"}
Sep 18 16:22:36.406077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.405920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"}
Sep 18 16:22:36.406077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.405958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"}
Sep 18 16:22:36.406077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.406000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"}
Sep 18 16:22:36.406077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.406034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"}
Sep 18 16:22:36.407186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.407157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn" event={"ID":"50ddce52-3943-4e3d-ac23-24bf7f669fd8","Type":"ContainerStarted","Data":"4b974ede0aec70f1c3ac81de4a01384fbd3a6558ce1cabd8cbdcf8359baf6820"}
Sep 18 16:22:36.546674 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.546638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:36.648394 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.648305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:36.692654 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:22:36.692605    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9170ef7_39a5_4934_8499_b2fc1fe927d0.slice/crio-4398f8ea17a9a2717d069181ed376b2f6bf6a1411c042a7afe93f6a65f4922e8 WatchSource:0}: Error finding container 4398f8ea17a9a2717d069181ed376b2f6bf6a1411c042a7afe93f6a65f4922e8: Status 404 returned error can't find the container with id 4398f8ea17a9a2717d069181ed376b2f6bf6a1411c042a7afe93f6a65f4922e8
Sep 18 16:22:36.695055 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.694976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"]
Sep 18 16:22:36.954729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:36.954648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:37.415042 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:37.414825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw" event={"ID":"b9170ef7-39a5-4934-8499-b2fc1fe927d0","Type":"ContainerStarted","Data":"4398f8ea17a9a2717d069181ed376b2f6bf6a1411c042a7afe93f6a65f4922e8"}
Sep 18 16:22:38.420340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.420303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw" event={"ID":"b9170ef7-39a5-4934-8499-b2fc1fe927d0","Type":"ContainerStarted","Data":"3e9dc483f428cdf92793c0a1a348814681c33dbda00929fca61587baf8cb388c"}
Sep 18 16:22:38.420800 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.420486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:38.424217 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.424190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerStarted","Data":"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"}
Sep 18 16:22:38.425683 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.425661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw"
Sep 18 16:22:38.425891 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.425868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn" event={"ID":"50ddce52-3943-4e3d-ac23-24bf7f669fd8","Type":"ContainerStarted","Data":"fbd361777b45da2cea16538962ea6cc77f5db475dd7928fbfc290144d58d8ecb"}
Sep 18 16:22:38.442455 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.442397    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-zv6kw" podStartSLOduration=1.842638024 podStartE2EDuration="3.44238129s" totalImagesPullingTime="1.599743266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:38.441358702 +0000 UTC m=+370.441619538" watchObservedRunningTime="2026-09-18 16:22:38.44238129 +0000 UTC m=+370.442642102"
Sep 18 16:22:38.462690 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.462644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn" podStartSLOduration=1.934225804 podStartE2EDuration="3.462630227s" totalImagesPullingTime="1.528404423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:38.461681723 +0000 UTC m=+370.461942549" watchObservedRunningTime="2026-09-18 16:22:38.462630227 +0000 UTC m=+370.462891038"
Sep 18 16:22:38.543265 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:38.543173    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.760707608 podStartE2EDuration="7.543156389s" totalImagesPullingTime="3.766719127s" totalInitContainerRuntime="15.729654ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:22:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:22:38.542066337 +0000 UTC m=+370.542327175" watchObservedRunningTime="2026-09-18 16:22:38.543156389 +0000 UTC m=+370.543417197"
Sep 18 16:22:39.465117 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:39.465088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:39.565554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:39.565526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:40.555794 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:40.555760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:40.656320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:40.656290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:43.527887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:43.527847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:43.596164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:43.596129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:44.053578 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:44.053532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:44.154090 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:44.154065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:44.254634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:44.254605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:46.218156 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:46.218122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:46.306855 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:46.306817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:46.407541 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:46.407510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:55.366732 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:55.366682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:55.366732 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:55.366755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:22:58.727924 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:58.727887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:58.829312 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:58.829280    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:22:58.892177 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:22:58.892144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:02.751900 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:02.751863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:02.813443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:02.813412    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:15.371774 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:15.371732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:23:15.375580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:15.375532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d586cdf54-5f8fn"
Sep 18 16:23:19.605990 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:19.605959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:19.665211 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:19.665183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:49.993749 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:49.993711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:50.195317 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:50.195284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:51.069070 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:23:51.069555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069307    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="alertmanager" containerID="cri-o://5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e" gracePeriod=120
Sep 18 16:23:51.069555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069391    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-metric" containerID="cri-o://dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba" gracePeriod=120
Sep 18 16:23:51.069555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069419    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy" containerID="cri-o://22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4" gracePeriod=120
Sep 18 16:23:51.069555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069414    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="config-reloader" containerID="cri-o://0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32" gracePeriod=120
Sep 18 16:23:51.069555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069391    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-web" containerID="cri-o://66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6" gracePeriod=120
Sep 18 16:23:51.069555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.069482    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="prom-label-proxy" containerID="cri-o://9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4" gracePeriod=120
Sep 18 16:23:51.113235 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:51.113199    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef632de_d4fa_4614_94c3_5f6c664b21d6.slice/crio-conmon-9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef632de_d4fa_4614_94c3_5f6c664b21d6.slice/crio-conmon-5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 16:23:51.681777 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681748    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4" exitCode=0
Sep 18 16:23:51.681777 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681771    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4" exitCode=0
Sep 18 16:23:51.681777 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681778    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32" exitCode=0
Sep 18 16:23:51.681777 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681783    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e" exitCode=0
Sep 18 16:23:51.682040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"}
Sep 18 16:23:51.682040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"}
Sep 18 16:23:51.682040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"}
Sep 18 16:23:51.682040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:51.681944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"}
Sep 18 16:23:52.308859 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.308833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.400259 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400233    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-main-tls\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqs2p\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-kube-api-access-mqs2p\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-volume\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-cluster-tls-config\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-web-config\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-metrics-client-ca\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-tls-assets\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400570    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-main-db\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.400811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-out\") pod \"aef632de-d4fa-4614-94c3-5f6c664b21d6\" (UID: \"aef632de-d4fa-4614-94c3-5f6c664b21d6\") "
Sep 18 16:23:52.401212 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.400811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:52.401212 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.401109    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:23:52.402344 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.402288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:23:52.403707 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.403676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.403987 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.403956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.404066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.404045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-volume" (OuterVolumeSpecName: "config-volume") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.404264 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.404227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-out" (OuterVolumeSpecName: "config-out") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:23:52.404838 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.404809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-kube-api-access-mqs2p" (OuterVolumeSpecName: "kube-api-access-mqs2p") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "kube-api-access-mqs2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:52.405216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.405184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.405340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.405323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:23:52.405340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.405328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.408093 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.408071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.413736 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.413711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-web-config" (OuterVolumeSpecName: "web-config") pod "aef632de-d4fa-4614-94c3-5f6c664b21d6" (UID: "aef632de-d4fa-4614-94c3-5f6c664b21d6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:23:52.502178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502129    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502173    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-main-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502184    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqs2p\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-kube-api-access-mqs2p\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502193    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-volume\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502201    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-cluster-tls-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502210    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502218    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-web-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502227    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aef632de-d4fa-4614-94c3-5f6c664b21d6-metrics-client-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502237    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502246    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aef632de-d4fa-4614-94c3-5f6c664b21d6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502254    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aef632de-d4fa-4614-94c3-5f6c664b21d6-tls-assets\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502262    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-alertmanager-main-db\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.502443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.502270    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aef632de-d4fa-4614-94c3-5f6c664b21d6-config-out\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:23:52.694287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694209    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba" exitCode=0
Sep 18 16:23:52.694287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694233    2563 generic.go:356] "Generic (PLEG): container finished" podID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerID="66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6" exitCode=0
Sep 18 16:23:52.694459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.694459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"}
Sep 18 16:23:52.694459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"}
Sep 18 16:23:52.694551 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aef632de-d4fa-4614-94c3-5f6c664b21d6","Type":"ContainerDied","Data":"7b98bb68cc6e532962cf1a6aa830daa9410889d0364600fcc73090288faa62a9"}
Sep 18 16:23:52.694551 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.694518    2563 scope.go:117] "RemoveContainer" containerID="9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"
Sep 18 16:23:52.704944 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.704925    2563 scope.go:117] "RemoveContainer" containerID="dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"
Sep 18 16:23:52.712363 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.712349    2563 scope.go:117] "RemoveContainer" containerID="22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"
Sep 18 16:23:52.718710 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.718693    2563 scope.go:117] "RemoveContainer" containerID="66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"
Sep 18 16:23:52.721000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.720982    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:23:52.725642 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.725622    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:23:52.725988 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.725970    2563 scope.go:117] "RemoveContainer" containerID="0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"
Sep 18 16:23:52.732642 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.732627    2563 scope.go:117] "RemoveContainer" containerID="5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"
Sep 18 16:23:52.739152 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.739136    2563 scope.go:117] "RemoveContainer" containerID="a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"
Sep 18 16:23:52.745663 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.745645    2563 scope.go:117] "RemoveContainer" containerID="9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"
Sep 18 16:23:52.745950 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.745931    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4\": container with ID starting with 9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4 not found: ID does not exist" containerID="9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"
Sep 18 16:23:52.745992 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.745959    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"} err="failed to get container status \"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4\": rpc error: code = NotFound desc = could not find container \"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4\": container with ID starting with 9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4 not found: ID does not exist"
Sep 18 16:23:52.746030 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.745990    2563 scope.go:117] "RemoveContainer" containerID="dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"
Sep 18 16:23:52.746233 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.746216    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba\": container with ID starting with dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba not found: ID does not exist" containerID="dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"
Sep 18 16:23:52.746280 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746238    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"} err="failed to get container status \"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba\": rpc error: code = NotFound desc = could not find container \"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba\": container with ID starting with dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba not found: ID does not exist"
Sep 18 16:23:52.746280 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746253    2563 scope.go:117] "RemoveContainer" containerID="22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"
Sep 18 16:23:52.746469 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.746454    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4\": container with ID starting with 22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4 not found: ID does not exist" containerID="22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"
Sep 18 16:23:52.746508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746473    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"} err="failed to get container status \"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4\": rpc error: code = NotFound desc = could not find container \"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4\": container with ID starting with 22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4 not found: ID does not exist"
Sep 18 16:23:52.746508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746487    2563 scope.go:117] "RemoveContainer" containerID="66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"
Sep 18 16:23:52.746719 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.746704    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6\": container with ID starting with 66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6 not found: ID does not exist" containerID="66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"
Sep 18 16:23:52.746770 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"} err="failed to get container status \"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6\": rpc error: code = NotFound desc = could not find container \"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6\": container with ID starting with 66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6 not found: ID does not exist"
Sep 18 16:23:52.746770 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746733    2563 scope.go:117] "RemoveContainer" containerID="0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"
Sep 18 16:23:52.746930 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.746916    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32\": container with ID starting with 0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32 not found: ID does not exist" containerID="0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"
Sep 18 16:23:52.746970 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746933    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"} err="failed to get container status \"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32\": rpc error: code = NotFound desc = could not find container \"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32\": container with ID starting with 0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32 not found: ID does not exist"
Sep 18 16:23:52.746970 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.746944    2563 scope.go:117] "RemoveContainer" containerID="5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"
Sep 18 16:23:52.747150 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.747134    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e\": container with ID starting with 5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e not found: ID does not exist" containerID="5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"
Sep 18 16:23:52.747189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747154    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"} err="failed to get container status \"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e\": rpc error: code = NotFound desc = could not find container \"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e\": container with ID starting with 5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e not found: ID does not exist"
Sep 18 16:23:52.747189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747166    2563 scope.go:117] "RemoveContainer" containerID="a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"
Sep 18 16:23:52.747360 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:23:52.747347    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62\": container with ID starting with a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62 not found: ID does not exist" containerID="a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"
Sep 18 16:23:52.747399 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747363    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"} err="failed to get container status \"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62\": rpc error: code = NotFound desc = could not find container \"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62\": container with ID starting with a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62 not found: ID does not exist"
Sep 18 16:23:52.747399 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747373    2563 scope.go:117] "RemoveContainer" containerID="9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"
Sep 18 16:23:52.747672 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747653    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4"} err="failed to get container status \"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4\": rpc error: code = NotFound desc = could not find container \"9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4\": container with ID starting with 9b446d76847c63b7f3aa57ac9c89ea2966390bc2d8de125056adb38dba1c91c4 not found: ID does not exist"
Sep 18 16:23:52.747740 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747673    2563 scope.go:117] "RemoveContainer" containerID="dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"
Sep 18 16:23:52.747877 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747862    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba"} err="failed to get container status \"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba\": rpc error: code = NotFound desc = could not find container \"dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba\": container with ID starting with dff9c96e37abcc472dce2f334059f1b9593c3bb4224fbee8071ea0de173299ba not found: ID does not exist"
Sep 18 16:23:52.747877 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.747875    2563 scope.go:117] "RemoveContainer" containerID="22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"
Sep 18 16:23:52.748108 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748093    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4"} err="failed to get container status \"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4\": rpc error: code = NotFound desc = could not find container \"22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4\": container with ID starting with 22d7983092e5fa61fa8fb8a8ca871310fc3e623a3af48f0e6f4bb01feef388c4 not found: ID does not exist"
Sep 18 16:23:52.748162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748107    2563 scope.go:117] "RemoveContainer" containerID="66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"
Sep 18 16:23:52.748316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748300    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6"} err="failed to get container status \"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6\": rpc error: code = NotFound desc = could not find container \"66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6\": container with ID starting with 66ffb29194837dcbf453c1cdf0aa27e1a1a2ee105318a00960a940d22fe542c6 not found: ID does not exist"
Sep 18 16:23:52.748316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748314    2563 scope.go:117] "RemoveContainer" containerID="0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"
Sep 18 16:23:52.748513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748494    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32"} err="failed to get container status \"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32\": rpc error: code = NotFound desc = could not find container \"0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32\": container with ID starting with 0877294eb385d7465691899cd4a6600adfe087d15bdf125c90caa709c2f18e32 not found: ID does not exist"
Sep 18 16:23:52.748604 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748515    2563 scope.go:117] "RemoveContainer" containerID="5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"
Sep 18 16:23:52.748797 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748782    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e"} err="failed to get container status \"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e\": rpc error: code = NotFound desc = could not find container \"5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e\": container with ID starting with 5effb2fc9890668d500adee932aced5d8761cca09434a436d393d0f1b47b4d0e not found: ID does not exist"
Sep 18 16:23:52.748797 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.748796    2563 scope.go:117] "RemoveContainer" containerID="a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"
Sep 18 16:23:52.749024 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.749006    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62"} err="failed to get container status \"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62\": rpc error: code = NotFound desc = could not find container \"a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62\": container with ID starting with a4fcde39c337eba66a408bc703b35a40a5816fbff0bae7dcd9968f9c86a3de62 not found: ID does not exist"
Sep 18 16:23:52.753688 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753668    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:23:52.753910 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753899    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="init-config-reloader"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753913    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="init-config-reloader"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753922    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-web"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-web"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753937    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753948    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="prom-label-proxy"
Sep 18 16:23:52.753956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753954    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="prom-label-proxy"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753962    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="config-reloader"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753967    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="config-reloader"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753975    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="alertmanager"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753980    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="alertmanager"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753987    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-metric"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.753992    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-metric"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.754031    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="alertmanager"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.754039    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-web"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.754045    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.754049    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="prom-label-proxy"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.754055    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="config-reloader"
Sep 18 16:23:52.754153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.754059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" containerName="kube-rbac-proxy-metric"
Sep 18 16:23:52.758728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.758714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.761285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 16:23:52.761368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 16:23:52.761368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 16:23:52.761737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vldj4\""
Sep 18 16:23:52.761737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 16:23:52.761737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 16:23:52.761935 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 16:23:52.761935 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.761784    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 16:23:52.762097 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.762079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 16:23:52.767631 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.767608    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 16:23:52.774644 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.774624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:23:52.905368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4669a34e-9a6a-4471-aced-6747c0f1cd77-config-out\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4669a34e-9a6a-4471-aced-6747c0f1cd77-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4669a34e-9a6a-4471-aced-6747c0f1cd77-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4669a34e-9a6a-4471-aced-6747c0f1cd77-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-config-volume\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905632 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-web-config\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hq8\" (UniqueName: \"kubernetes.io/projected/4669a34e-9a6a-4471-aced-6747c0f1cd77-kube-api-access-k7hq8\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:52.905931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:52.905757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4669a34e-9a6a-4471-aced-6747c0f1cd77-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.006792 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.006792 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4669a34e-9a6a-4471-aced-6747c0f1cd77-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.006792 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.006792 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4669a34e-9a6a-4471-aced-6747c0f1cd77-config-out\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4669a34e-9a6a-4471-aced-6747c0f1cd77-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4669a34e-9a6a-4471-aced-6747c0f1cd77-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4669a34e-9a6a-4471-aced-6747c0f1cd77-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.006905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.007003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-config-volume\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.007053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.007084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-web-config\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007123 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.007121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.007488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.007156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hq8\" (UniqueName: \"kubernetes.io/projected/4669a34e-9a6a-4471-aced-6747c0f1cd77-kube-api-access-k7hq8\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.009255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.007996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4669a34e-9a6a-4471-aced-6747c0f1cd77-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.009255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.008941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4669a34e-9a6a-4471-aced-6747c0f1cd77-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.009255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.009174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4669a34e-9a6a-4471-aced-6747c0f1cd77-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.010185 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.010160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.010500 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.010476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4669a34e-9a6a-4471-aced-6747c0f1cd77-config-out\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.010642 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.010613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4669a34e-9a6a-4471-aced-6747c0f1cd77-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.010754 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.010708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-web-config\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.010980 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.010957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-config-volume\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.011055 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.010957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.011104 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.011048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.011372 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.011355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.012400 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.012384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4669a34e-9a6a-4471-aced-6747c0f1cd77-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.015133 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.015117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hq8\" (UniqueName: \"kubernetes.io/projected/4669a34e-9a6a-4471-aced-6747c0f1cd77-kube-api-access-k7hq8\") pod \"alertmanager-main-0\" (UID: \"4669a34e-9a6a-4471-aced-6747c0f1cd77\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.067970 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.067933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 16:23:53.350686 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.350632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 16:23:53.352759 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:23:53.352731    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4669a34e_9a6a_4471_aced_6747c0f1cd77.slice/crio-3fb6f84050ec7e2d4406557c9289a0fa55c98982a039e433a94f715abf5c1d89 WatchSource:0}: Error finding container 3fb6f84050ec7e2d4406557c9289a0fa55c98982a039e433a94f715abf5c1d89: Status 404 returned error can't find the container with id 3fb6f84050ec7e2d4406557c9289a0fa55c98982a039e433a94f715abf5c1d89
Sep 18 16:23:53.699842 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.699808    2563 generic.go:356] "Generic (PLEG): container finished" podID="4669a34e-9a6a-4471-aced-6747c0f1cd77" containerID="eea9b5fba754677fc2bfcda4f11f6fe5e2eeee0847207f787e2b840c5d1ab6d6" exitCode=0
Sep 18 16:23:53.700040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.699855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerDied","Data":"eea9b5fba754677fc2bfcda4f11f6fe5e2eeee0847207f787e2b840c5d1ab6d6"}
Sep 18 16:23:53.700040 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:53.699903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"3fb6f84050ec7e2d4406557c9289a0fa55c98982a039e433a94f715abf5c1d89"}
Sep 18 16:23:54.217291 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.217209    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef632de-d4fa-4614-94c3-5f6c664b21d6" path="/var/lib/kubelet/pods/aef632de-d4fa-4614-94c3-5f6c664b21d6/volumes"
Sep 18 16:23:54.707147 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.707117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"bb16028b2c3ee4770c7515276ea7eda490863046fb2fcc3ca8049e4b2c046da8"}
Sep 18 16:23:54.707513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.707185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"ff23db2ff2fbf8163c2f6227f7c12e4f74199e02338447deeb3e9522989e2821"}
Sep 18 16:23:54.707513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.707214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"d80556ead0cf0f7cc9718153e3745f6b5870a484d91a50252fecc461e969911c"}
Sep 18 16:23:54.707513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.707252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"86ef301f25757edb752cbeb781e48383ece7802e7bee5054d6edc70bfc1896e2"}
Sep 18 16:23:54.707513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.707280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"eb447bd0a4fc22c0f2fda392461d5cd21f0e56044e0673bbfcef2b888f69138d"}
Sep 18 16:23:54.707513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.707303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4669a34e-9a6a-4471-aced-6747c0f1cd77","Type":"ContainerStarted","Data":"b43d1ea9033027a1ac151a006a29629239c03044bd8aa124375ad24b96322b9a"}
Sep 18 16:23:54.733410 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:54.733367    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.714445959 podStartE2EDuration="2.733352619s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.90666ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:54.732960804 +0000 UTC m=+446.733221626" watchObservedRunningTime="2026-09-18 16:23:54.733352619 +0000 UTC m=+446.733613430"
Sep 18 16:23:55.098781 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.098696    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"]
Sep 18 16:23:55.102222 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.102206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.104376 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.104355    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 16:23:55.104576 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.104547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 16:23:55.104708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.104685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 16:23:55.104795 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.104712    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zgnss\""
Sep 18 16:23:55.104905 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.104895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 16:23:55.105076 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.105060    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 16:23:55.109353 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.109335    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 16:23:55.114907 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.114888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"]
Sep 18 16:23:55.224856 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.224820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-serving-certs-ca-bundle\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.224856 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.224860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-secret-telemeter-client\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.225101 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.224948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-federate-client-tls\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.225101 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.224981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.225101 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.225006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-metrics-client-ca\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.225101 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.225062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rw4\" (UniqueName: \"kubernetes.io/projected/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-kube-api-access-m9rw4\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.225101 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.225100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-telemeter-client-tls\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.225300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.225123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.325808 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-serving-certs-ca-bundle\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.325948 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-secret-telemeter-client\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.325948 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-federate-client-tls\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.325948 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.325948 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-metrics-client-ca\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.325948 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rw4\" (UniqueName: \"kubernetes.io/projected/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-kube-api-access-m9rw4\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.326193 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-telemeter-client-tls\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.326193 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.325975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.326673 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.326646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-metrics-client-ca\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.326855 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.326832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-serving-certs-ca-bundle\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.327080 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.327057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.328587 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.328543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.328672 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.328594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-secret-telemeter-client\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.329000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.328983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-telemeter-client-tls\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.329058 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.329041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-federate-client-tls\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.333826 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.333809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rw4\" (UniqueName: \"kubernetes.io/projected/7e17ec68-5d31-4477-a8c8-72ee26b3d0b2-kube-api-access-m9rw4\") pod \"telemeter-client-76f4bd6868-ncgd2\" (UID: \"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2\") " pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.412912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.412877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"
Sep 18 16:23:55.687572 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.687518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76f4bd6868-ncgd2"]
Sep 18 16:23:55.689981 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:23:55.689950    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e17ec68_5d31_4477_a8c8_72ee26b3d0b2.slice/crio-9061cad77473ef774352da206145fd5a5014b8299798063343a4b3bf4e00dfab WatchSource:0}: Error finding container 9061cad77473ef774352da206145fd5a5014b8299798063343a4b3bf4e00dfab: Status 404 returned error can't find the container with id 9061cad77473ef774352da206145fd5a5014b8299798063343a4b3bf4e00dfab
Sep 18 16:23:55.713187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:55.713161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2" event={"ID":"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2","Type":"ContainerStarted","Data":"9061cad77473ef774352da206145fd5a5014b8299798063343a4b3bf4e00dfab"}
Sep 18 16:23:57.723436 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:57.723351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2" event={"ID":"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2","Type":"ContainerStarted","Data":"55612c58174a7894b2b1bd56a3bb51cb002a46f6b583f50e509b9b5dc10b5d79"}
Sep 18 16:23:57.723436 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:57.723435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2" event={"ID":"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2","Type":"ContainerStarted","Data":"209d8556df487c733b88a6434ee0843571bfe806b5b5625e21e4713dfaad4d3d"}
Sep 18 16:23:57.723885 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:57.723456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2" event={"ID":"7e17ec68-5d31-4477-a8c8-72ee26b3d0b2","Type":"ContainerStarted","Data":"90d3d829d0be6118a21e3a59e82d9d652571b097ae5c32870076a0cd9c3cb397"}
Sep 18 16:23:57.745877 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:57.745830    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76f4bd6868-ncgd2" podStartSLOduration=1.019025842 podStartE2EDuration="2.745813125s" totalImagesPullingTime="1.726787283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:57.74386941 +0000 UTC m=+449.744130232" watchObservedRunningTime="2026-09-18 16:23:57.745813125 +0000 UTC m=+449.746073936"
Sep 18 16:23:57.892313 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:57.892272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:57.952139 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:57.952107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:23:58.604438 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:23:58.604408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:08.837877 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:24:08.837841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:08.941668 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:24:08.941634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:09.039195 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:24:09.039170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:24:39.150080 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:24:39.150031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:05.262478 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:05.262443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:05.360173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:05.360135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:06.114109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:06.114077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:06.226654 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:06.226621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:17.113799 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:17.113716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:17.214088 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:17.214060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:17.282337 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:17.282307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:32.775582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:32.775522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:34.130748 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:34.130714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:38.730192 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:38.730162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:38.826139 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:38.826113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:41.561703 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:41.561667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:41.623736 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:41.623711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:44.129732 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:44.129703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:57.329501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:57.329461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:25:57.430170 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:25:57.430140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:03.322253 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:03.322211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:14.281620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:14.281580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:14.384701 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:14.384666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:14.454034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:14.453993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:20.447127 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.447095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:20.543419 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.543376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"]
Sep 18 16:26:20.546473 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.546457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.549759 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.549737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 16:26:20.549759 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.549750    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 16:26:20.549999 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.549775    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:26:20.549999 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.549738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 16:26:20.554913 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.554893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:20.558407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.558384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"]
Sep 18 16:26:20.564021 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.564001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"]
Sep 18 16:26:20.566907 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.566894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.569185 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.569166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2phs4\""
Sep 18 16:26:20.569291 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.569276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 16:26:20.584581 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.583448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"]
Sep 18 16:26:20.633174 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.633137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:20.733109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.733030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9lz\" (UniqueName: \"kubernetes.io/projected/a3d37646-ea5b-478e-bc16-56cf3c1660b3-kube-api-access-7x9lz\") pod \"managed-serviceaccount-addon-agent-69d9875cbb-sdjdg\" (UID: \"a3d37646-ea5b-478e-bc16-56cf3c1660b3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.733109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.733082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.733109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.733103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpqb\" (UniqueName: \"kubernetes.io/projected/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-kube-api-access-ljpqb\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.733305 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.733156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3d37646-ea5b-478e-bc16-56cf3c1660b3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9875cbb-sdjdg\" (UID: \"a3d37646-ea5b-478e-bc16-56cf3c1660b3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.733305 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.733190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-tmp\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.833806 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.833769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-tmp\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.833880 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.833833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9lz\" (UniqueName: \"kubernetes.io/projected/a3d37646-ea5b-478e-bc16-56cf3c1660b3-kube-api-access-7x9lz\") pod \"managed-serviceaccount-addon-agent-69d9875cbb-sdjdg\" (UID: \"a3d37646-ea5b-478e-bc16-56cf3c1660b3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.833880 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.833872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.833960 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.833890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpqb\" (UniqueName: \"kubernetes.io/projected/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-kube-api-access-ljpqb\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.833960 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.833913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3d37646-ea5b-478e-bc16-56cf3c1660b3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9875cbb-sdjdg\" (UID: \"a3d37646-ea5b-478e-bc16-56cf3c1660b3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.834186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.834164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-tmp\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.836626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.836596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.836626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.836617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3d37646-ea5b-478e-bc16-56cf3c1660b3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9875cbb-sdjdg\" (UID: \"a3d37646-ea5b-478e-bc16-56cf3c1660b3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.842990 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.842970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9lz\" (UniqueName: \"kubernetes.io/projected/a3d37646-ea5b-478e-bc16-56cf3c1660b3-kube-api-access-7x9lz\") pod \"managed-serviceaccount-addon-agent-69d9875cbb-sdjdg\" (UID: \"a3d37646-ea5b-478e-bc16-56cf3c1660b3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:20.843196 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.843175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpqb\" (UniqueName: \"kubernetes.io/projected/a3b607e2-78fc-40c4-8ab7-e6097419bcf5-kube-api-access-ljpqb\") pod \"klusterlet-addon-workmgr-64cf4f47bf-mjp84\" (UID: \"a3b607e2-78fc-40c4-8ab7-e6097419bcf5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.855809 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.855790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:20.887313 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:20.887286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"
Sep 18 16:26:21.128898 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:21.128874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"]
Sep 18 16:26:21.131187 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:26:21.131157    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b607e2_78fc_40c4_8ab7_e6097419bcf5.slice/crio-ef6c027d2d100ddfb7e66b5197594fc7f5aaae0a49fed1ea382f5b33e5cec0f7 WatchSource:0}: Error finding container ef6c027d2d100ddfb7e66b5197594fc7f5aaae0a49fed1ea382f5b33e5cec0f7: Status 404 returned error can't find the container with id ef6c027d2d100ddfb7e66b5197594fc7f5aaae0a49fed1ea382f5b33e5cec0f7
Sep 18 16:26:21.161718 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:21.161696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg"]
Sep 18 16:26:21.164014 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:26:21.163987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d37646_ea5b_478e_bc16_56cf3c1660b3.slice/crio-0988509e1c4fee344a56f5c6ce48dca98f6838af1429977f9f83c6899b12170c WatchSource:0}: Error finding container 0988509e1c4fee344a56f5c6ce48dca98f6838af1429977f9f83c6899b12170c: Status 404 returned error can't find the container with id 0988509e1c4fee344a56f5c6ce48dca98f6838af1429977f9f83c6899b12170c
Sep 18 16:26:21.238104 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:21.238071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84" event={"ID":"a3b607e2-78fc-40c4-8ab7-e6097419bcf5","Type":"ContainerStarted","Data":"ef6c027d2d100ddfb7e66b5197594fc7f5aaae0a49fed1ea382f5b33e5cec0f7"}
Sep 18 16:26:21.239244 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:21.239221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg" event={"ID":"a3d37646-ea5b-478e-bc16-56cf3c1660b3","Type":"ContainerStarted","Data":"0988509e1c4fee344a56f5c6ce48dca98f6838af1429977f9f83c6899b12170c"}
Sep 18 16:26:22.259413 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:22.259373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:22.356555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:22.356527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:22.456363 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:22.456334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:26:27.266788 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:27.266749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84" event={"ID":"a3b607e2-78fc-40c4-8ab7-e6097419bcf5","Type":"ContainerStarted","Data":"2cf06ad2024b8c8216210b9bbffdb00aa3eb6f24997f335b8547a47abbe8e7f8"}
Sep 18 16:26:27.267264 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:27.266904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:27.268384 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:27.268326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg" event={"ID":"a3d37646-ea5b-478e-bc16-56cf3c1660b3","Type":"ContainerStarted","Data":"0db5cc1ef091a4861ad5e8f04a68930562c069cfc758debb7fc415ea396433e0"}
Sep 18 16:26:27.269025 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:27.269008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84"
Sep 18 16:26:27.282841 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:27.282796    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cf4f47bf-mjp84" podStartSLOduration=1.949249541 podStartE2EDuration="7.282782587s" totalImagesPullingTime="5.333533046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:27.281711784 +0000 UTC m=+599.281972615" watchObservedRunningTime="2026-09-18 16:26:27.282782587 +0000 UTC m=+599.283043398"
Sep 18 16:26:27.298002 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:27.297957    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9875cbb-sdjdg" podStartSLOduration=2.007895211 podStartE2EDuration="7.2979435s" totalImagesPullingTime="5.290048289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:27.296180524 +0000 UTC m=+599.296441335" watchObservedRunningTime="2026-09-18 16:26:27.2979435 +0000 UTC m=+599.298204310"
Sep 18 16:26:30.100195 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:30.100158    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:26:30.101606 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:26:30.101576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:27:36.192923 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:36.192889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:36.305931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:36.305893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:41.225730 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.225688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:41.517347 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.517271    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7h587"]
Sep 18 16:27:41.519374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.519359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.522238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.522216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 16:27:41.522968 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.522950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 16:27:41.523052 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.522962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zjtn5\""
Sep 18 16:27:41.523052 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.522962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 16:27:41.523230 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.523214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 16:27:41.523320 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.523254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 16:27:41.535923 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.535897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7h587"]
Sep 18 16:27:41.536184 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.536164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:41.558962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.558930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fb7c7510-533b-459e-ab26-afd12e004022-cabundle0\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.559083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.558975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmgt\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-kube-api-access-xvmgt\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.559083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.559012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-certificates\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.611554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.611530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:41.659959 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.659935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fb7c7510-533b-459e-ab26-afd12e004022-cabundle0\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.660068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.659974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmgt\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-kube-api-access-xvmgt\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.660068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.660055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-certificates\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.660168 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:27:41.660156    2563 secret.go:281] references non-existent secret key: ca.crt
Sep 18 16:27:41.660203 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:27:41.660176    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 16:27:41.660203 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:27:41.660187    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-7h587: references non-existent secret key: ca.crt
Sep 18 16:27:41.660267 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:27:41.660240    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-certificates podName:fb7c7510-533b-459e-ab26-afd12e004022 nodeName:}" failed. No retries permitted until 2026-09-18 16:27:42.160223022 +0000 UTC m=+674.160483825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-certificates") pod "keda-operator-579c8958f7-7h587" (UID: "fb7c7510-533b-459e-ab26-afd12e004022") : references non-existent secret key: ca.crt
Sep 18 16:27:41.660551 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.660534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/fb7c7510-533b-459e-ab26-afd12e004022-cabundle0\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.668801 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.668772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmgt\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-kube-api-access-xvmgt\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:41.722618 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.722583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:41.949495 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:41.949438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:42.163955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.163927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-certificates\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:42.166434 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.166407    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fb7c7510-533b-459e-ab26-afd12e004022-certificates\") pod \"keda-operator-579c8958f7-7h587\" (UID: \"fb7c7510-533b-459e-ab26-afd12e004022\") " pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:42.280187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.280114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:42.287881 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.287844    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nrgq8"]
Sep 18 16:27:42.290433 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.290413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.293337 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.293314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 16:27:42.302436 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.302415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nrgq8"]
Sep 18 16:27:42.365238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.365203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmk5\" (UniqueName: \"kubernetes.io/projected/697922e2-d584-4d8c-9a0e-74fda7192a54-kube-api-access-tnmk5\") pod \"keda-admission-6c4d864b86-nrgq8\" (UID: \"697922e2-d584-4d8c-9a0e-74fda7192a54\") " pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.365385 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.365286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/697922e2-d584-4d8c-9a0e-74fda7192a54-certificates\") pod \"keda-admission-6c4d864b86-nrgq8\" (UID: \"697922e2-d584-4d8c-9a0e-74fda7192a54\") " pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.380710 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.380681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:42.429305 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.429278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:42.466735 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.466700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/697922e2-d584-4d8c-9a0e-74fda7192a54-certificates\") pod \"keda-admission-6c4d864b86-nrgq8\" (UID: \"697922e2-d584-4d8c-9a0e-74fda7192a54\") " pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.466910 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.466776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmk5\" (UniqueName: \"kubernetes.io/projected/697922e2-d584-4d8c-9a0e-74fda7192a54-kube-api-access-tnmk5\") pod \"keda-admission-6c4d864b86-nrgq8\" (UID: \"697922e2-d584-4d8c-9a0e-74fda7192a54\") " pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.469619 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.469591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/697922e2-d584-4d8c-9a0e-74fda7192a54-certificates\") pod \"keda-admission-6c4d864b86-nrgq8\" (UID: \"697922e2-d584-4d8c-9a0e-74fda7192a54\") " pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.475154 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.475084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmk5\" (UniqueName: \"kubernetes.io/projected/697922e2-d584-4d8c-9a0e-74fda7192a54-kube-api-access-tnmk5\") pod \"keda-admission-6c4d864b86-nrgq8\" (UID: \"697922e2-d584-4d8c-9a0e-74fda7192a54\") " pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.477727 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.477703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:42.576956 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.576878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:42.600747 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.600718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:42.698940 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.698919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7h587"]
Sep 18 16:27:42.701586 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:27:42.701544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7c7510_533b_459e_ab26_afd12e004022.slice/crio-1a190fdef28516eed263b94e28a00d87cbc131526f6ee08fdfc42c6b1e558f60 WatchSource:0}: Error finding container 1a190fdef28516eed263b94e28a00d87cbc131526f6ee08fdfc42c6b1e558f60: Status 404 returned error can't find the container with id 1a190fdef28516eed263b94e28a00d87cbc131526f6ee08fdfc42c6b1e558f60
Sep 18 16:27:42.703069 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.703049    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:27:42.778240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.778211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:42.868616 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.868476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nrgq8"]
Sep 18 16:27:42.871075 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:27:42.871052    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697922e2_d584_4d8c_9a0e_74fda7192a54.slice/crio-d4b10d8bffc38907cfa066ce6b35d35cd51ab17701f75fc4e98a21d94042fd19 WatchSource:0}: Error finding container d4b10d8bffc38907cfa066ce6b35d35cd51ab17701f75fc4e98a21d94042fd19: Status 404 returned error can't find the container with id d4b10d8bffc38907cfa066ce6b35d35cd51ab17701f75fc4e98a21d94042fd19
Sep 18 16:27:42.936910 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:42.936881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:43.031850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:43.031809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:27:43.552289 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:43.552250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8" event={"ID":"697922e2-d584-4d8c-9a0e-74fda7192a54","Type":"ContainerStarted","Data":"d4b10d8bffc38907cfa066ce6b35d35cd51ab17701f75fc4e98a21d94042fd19"}
Sep 18 16:27:43.553358 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:43.553337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7h587" event={"ID":"fb7c7510-533b-459e-ab26-afd12e004022","Type":"ContainerStarted","Data":"1a190fdef28516eed263b94e28a00d87cbc131526f6ee08fdfc42c6b1e558f60"}
Sep 18 16:27:47.571729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:47.571692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7h587" event={"ID":"fb7c7510-533b-459e-ab26-afd12e004022","Type":"ContainerStarted","Data":"8d8221783b0121dc9882d44213ce021b4672b5c61133f68def3922d307f9fa68"}
Sep 18 16:27:47.572180 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:47.571802    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:27:47.573250 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:47.573225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8" event={"ID":"697922e2-d584-4d8c-9a0e-74fda7192a54","Type":"ContainerStarted","Data":"5a99020afc7d9d59a8598b2cfb7885e25071d5c9245decbf15f415780ccdb9ad"}
Sep 18 16:27:47.573379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:47.573357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:27:47.590252 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:47.590214    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-7h587" podStartSLOduration=2.680187369 podStartE2EDuration="6.590201861s" totalImagesPullingTime="3.910014492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:47.588893849 +0000 UTC m=+679.589154661" watchObservedRunningTime="2026-09-18 16:27:47.590201861 +0000 UTC m=+679.590462663"
Sep 18 16:27:47.608311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:27:47.608266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8" podStartSLOduration=1.8758899169999999 podStartE2EDuration="5.608253102s" totalImagesPullingTime="3.732363185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:47.606626201 +0000 UTC m=+679.606887024" watchObservedRunningTime="2026-09-18 16:27:47.608253102 +0000 UTC m=+679.608513912"
Sep 18 16:28:08.580076 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:08.580037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-7h587"
Sep 18 16:28:08.580618 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:08.580154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-nrgq8"
Sep 18 16:28:25.814323 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:25.814285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:25.876721 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:25.876680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:34.281695 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:34.281656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:34.345220 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:34.345190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:28:45.475861 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:28:45.475826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:00.398921 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:00.398879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:00.488098 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:00.488068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:00.591748 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:00.591711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:00.691952 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:00.691859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:01.097846 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:01.097756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:01.196441 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:01.196410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:29:01.264214 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:29:01.264179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:18.398331 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.398301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:18.459927 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.459897    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-6xf7d"]
Sep 18 16:30:18.461934 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.461919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.464763 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.464737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 16:30:18.464896 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.464838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 16:30:18.465655 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.465634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-v8tqb\""
Sep 18 16:30:18.465745 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.465672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 16:30:18.465783 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.465743    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 16:30:18.479852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.479830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-6xf7d"]
Sep 18 16:30:18.483480 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.483457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:18.624668 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.624639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmr8\" (UniqueName: \"kubernetes.io/projected/8c02100f-6f28-49ed-97ce-023a29a93949-kube-api-access-gxmr8\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.624826 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.624680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-cert\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.624826 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.624714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-metrics-certs\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.710330 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.710246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:18.725357 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.725335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-cert\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.725445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.725372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-metrics-certs\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.725494 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:30:18.725476    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 16:30:18.725530 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.725500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmr8\" (UniqueName: \"kubernetes.io/projected/8c02100f-6f28-49ed-97ce-023a29a93949-kube-api-access-gxmr8\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.725579 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:30:18.725543    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-cert podName:8c02100f-6f28-49ed-97ce-023a29a93949 nodeName:}" failed. No retries permitted until 2026-09-18 16:30:19.225528819 +0000 UTC m=+831.225789631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-cert") pod "odh-model-controller-7b966d7984-6xf7d" (UID: "8c02100f-6f28-49ed-97ce-023a29a93949") : secret "odh-model-controller-webhook-cert" not found
Sep 18 16:30:18.728028 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.728007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-metrics-certs\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.734807 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.734785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmr8\" (UniqueName: \"kubernetes.io/projected/8c02100f-6f28-49ed-97ce-023a29a93949-kube-api-access-gxmr8\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:18.815720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:18.815689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:19.020848 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:19.020770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:19.229312 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:19.229276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-cert\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:19.231814 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:19.231795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c02100f-6f28-49ed-97ce-023a29a93949-cert\") pod \"odh-model-controller-7b966d7984-6xf7d\" (UID: \"8c02100f-6f28-49ed-97ce-023a29a93949\") " pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:19.371742 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:19.371644    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:19.648622 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:19.648533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-6xf7d"]
Sep 18 16:30:19.651187 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:30:19.651157    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c02100f_6f28_49ed_97ce_023a29a93949.slice/crio-87094f354388244b9c7db0229166fd63482e7a8d23d1e82c87484fafc1f5b61a WatchSource:0}: Error finding container 87094f354388244b9c7db0229166fd63482e7a8d23d1e82c87484fafc1f5b61a: Status 404 returned error can't find the container with id 87094f354388244b9c7db0229166fd63482e7a8d23d1e82c87484fafc1f5b61a
Sep 18 16:30:20.035738 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:20.035641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:20.099206 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:20.099173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:30:20.159138 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:20.159102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-6xf7d" event={"ID":"8c02100f-6f28-49ed-97ce-023a29a93949","Type":"ContainerStarted","Data":"87094f354388244b9c7db0229166fd63482e7a8d23d1e82c87484fafc1f5b61a"}
Sep 18 16:30:22.169142 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:22.169107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-6xf7d" event={"ID":"8c02100f-6f28-49ed-97ce-023a29a93949","Type":"ContainerStarted","Data":"07152ad27284f61d08382e7f1748b6c2440a3821d09bc62c9932763361862ee2"}
Sep 18 16:30:22.169547 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:22.169236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:22.210699 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:22.210644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-6xf7d" podStartSLOduration=1.764642813 podStartE2EDuration="4.21062702s" totalImagesPullingTime="2.445984207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:22.209824387 +0000 UTC m=+834.210085198" watchObservedRunningTime="2026-09-18 16:30:22.21062702 +0000 UTC m=+834.210887830"
Sep 18 16:30:33.175106 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:33.175076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-6xf7d"
Sep 18 16:30:52.292895 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:30:52.292853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:30.127059 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:31:30.127031    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:31:30.127535 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:31:30.127279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:31:54.526171 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:31:54.526138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:54.627203 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:31:54.627171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:57.323227 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:31:57.323187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:31:57.424786 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:31:57.424745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:06.575274 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.575238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:06.816942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.816912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:06.866177 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.866091    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"]
Sep 18 16:32:06.870110 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.870090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:06.873694 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.873671    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:32:06.874375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.874353    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:32:06.874466 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.874435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:32:06.874466 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.874453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:32:06.874594 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.874470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 16:32:06.874659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.874617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 16:32:06.877873 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.877857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:32:06.888386 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.888361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"]
Sep 18 16:32:06.930166 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.930135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:06.947723 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.947691    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"]
Sep 18 16:32:06.951237 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.951219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:06.954650 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.954635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-3f0a6-predictor-serving-cert\""
Sep 18 16:32:06.954748 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.954721    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-3f0a6-kube-rbac-proxy-sar-config\""
Sep 18 16:32:06.981584 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:06.981530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"]
Sep 18 16:32:07.005572 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.005536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.005714 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.005622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.005714 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.005664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.005802 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.005728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.005802 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.005788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld4c\" (UniqueName: \"kubernetes.io/projected/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-api-access-qld4c\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.103575 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.103531    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:07.106706 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld4c\" (UniqueName: \"kubernetes.io/projected/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-api-access-qld4c\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.106819 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.106819 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.106819 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.106964 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.106964 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ws68\" (UniqueName: \"kubernetes.io/projected/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-api-access-7ws68\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.106964 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.106916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.107115 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.107042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.107115 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.107077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.107223 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.107198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.107456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.107439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.107541 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.107494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.107634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.107600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.109686 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.109664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.117225 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.117174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld4c\" (UniqueName: \"kubernetes.io/projected/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-api-access-qld4c\") pod \"isvc-canary-rollback-predictor-7f7b644c9-rdm8j\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.180386 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.180356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:07.207638 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.207597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.207795 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.207731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.207795 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.207760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ws68\" (UniqueName: \"kubernetes.io/projected/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-api-access-7ws68\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.207795 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.207791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.207971 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.207835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.208398 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.208255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.208492 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.208450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.208611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.208588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.210681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.210658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-proxy-tls\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.221840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.221806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ws68\" (UniqueName: \"kubernetes.io/projected/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-api-access-7ws68\") pod \"isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.249996 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.249969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:07.260965 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.260941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:07.465797 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.465764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"]
Sep 18 16:32:07.471309 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:32:07.471287    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9c7bc7_e80b_4d5a_90c8_47c0f5261103.slice/crio-fc3e1db9ed3d7a024b75ff90f449de2717c39a5d8aef2c78d4f8d9382de0f9dc WatchSource:0}: Error finding container fc3e1db9ed3d7a024b75ff90f449de2717c39a5d8aef2c78d4f8d9382de0f9dc: Status 404 returned error can't find the container with id fc3e1db9ed3d7a024b75ff90f449de2717c39a5d8aef2c78d4f8d9382de0f9dc
Sep 18 16:32:07.550712 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.550681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"]
Sep 18 16:32:07.554135 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:32:07.554108    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ec894f_26e8_4f3f_a3c7_339bb7bb07d5.slice/crio-6f431dd860a42598df70f4c16d7d45670be9fc80f248f8a7f3bb237a0a92dcb5 WatchSource:0}: Error finding container 6f431dd860a42598df70f4c16d7d45670be9fc80f248f8a7f3bb237a0a92dcb5: Status 404 returned error can't find the container with id 6f431dd860a42598df70f4c16d7d45670be9fc80f248f8a7f3bb237a0a92dcb5
Sep 18 16:32:07.572894 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.572865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerStarted","Data":"fc3e1db9ed3d7a024b75ff90f449de2717c39a5d8aef2c78d4f8d9382de0f9dc"}
Sep 18 16:32:07.574057 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:07.574039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerStarted","Data":"6f431dd860a42598df70f4c16d7d45670be9fc80f248f8a7f3bb237a0a92dcb5"}
Sep 18 16:32:12.607540 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:12.607488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerStarted","Data":"95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4"}
Sep 18 16:32:12.609179 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:12.609152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerStarted","Data":"911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083"}
Sep 18 16:32:13.671515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:13.671467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:13.775153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:13.775119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:32:15.624242 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:15.624205    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerID="911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083" exitCode=0
Sep 18 16:32:15.624242 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:15.624230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerDied","Data":"911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083"}
Sep 18 16:32:15.626116 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:15.626048    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerID="95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4" exitCode=0
Sep 18 16:32:15.626182 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:15.626122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerDied","Data":"95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4"}
Sep 18 16:32:32.732234 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:32.732137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerStarted","Data":"fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8"}
Sep 18 16:32:32.733996 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:32.733971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerStarted","Data":"a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237"}
Sep 18 16:32:35.752699 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:35.752660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerStarted","Data":"c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8"}
Sep 18 16:32:35.753159 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:35.752959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:35.753159 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:35.753006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:35.754716 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:35.754684    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:32:35.755056 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:35.755021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerStarted","Data":"6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0"}
Sep 18 16:32:35.772677 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:35.772599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podStartSLOduration=2.45031977 podStartE2EDuration="29.772582126s" totalImagesPullingTime="23.662870607s" totalInitContainerRuntime="3.659391749s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:35.77219364 +0000 UTC m=+967.772454483" watchObservedRunningTime="2026-09-18 16:32:35.772582126 +0000 UTC m=+967.772842934"
Sep 18 16:32:36.759945 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:36.759905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:32:38.771570 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.771469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerStarted","Data":"e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918"}
Sep 18 16:32:38.771985 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.771697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:38.771985 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.771755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:38.771985 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.771786    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:38.773491 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.773459    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:32:38.774154 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.774129    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:32:38.794852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:38.794807    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podStartSLOduration=2.918746899 podStartE2EDuration="32.794788607s" totalImagesPullingTime="26.176612376s" totalInitContainerRuntime="3.699429332s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:06 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:38.792575463 +0000 UTC m=+970.792836283" watchObservedRunningTime="2026-09-18 16:32:38.794788607 +0000 UTC m=+970.795049414"
Sep 18 16:32:39.776120 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:39.776076    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:32:39.776610 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:39.776548    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:32:41.765505 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:41.765474    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:32:41.766287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:41.766253    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:32:44.780421 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:44.780391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:32:44.781066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:44.781029    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:32:44.781209 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:44.781184    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:32:51.766290 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:51.766240    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:32:54.781819 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:54.781773    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:32:54.782288 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:32:54.782206    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:33:01.767154 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:01.767110    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:33:02.603343 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:02.603312    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:04.781746 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:04.781705    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:33:04.782170 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:04.782120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:33:11.766460 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:11.766421    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:33:14.781554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:14.781506    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:33:14.782046 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:14.781951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:33:21.767232 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:21.767185    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:33:24.781794 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:24.781750    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:33:24.782400 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:24.782368    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:33:31.766383 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:31.766336    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:33:34.781756 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:34.781697    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:33:34.782250 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:34.782223    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:33:41.766811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:41.766772    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:33:44.781132 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:44.781079    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:33:44.781634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:44.781364    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:33:48.488017 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:48.487967    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"]
Sep 18 16:33:48.488502 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:48.488275    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" containerID="cri-o://fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8" gracePeriod=30
Sep 18 16:33:48.488502 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:48.488383    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kube-rbac-proxy" containerID="cri-o://c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8" gracePeriod=30
Sep 18 16:33:48.488715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:48.488689    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 16:33:49.069887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:49.069853    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerID="c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8" exitCode=2
Sep 18 16:33:49.070059 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:49.069915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerDied","Data":"c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8"}
Sep 18 16:33:51.761157 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:51.761112    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 18 16:33:54.534793 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.534740    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:33:54.554767 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.554732    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-proxy-tls\") pod \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") "
Sep 18 16:33:54.554887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.554793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qld4c\" (UniqueName: \"kubernetes.io/projected/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-api-access-qld4c\") pod \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") "
Sep 18 16:33:54.554887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.554822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-cabundle-cert\") pod \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") "
Sep 18 16:33:54.554887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.554846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-rbac-proxy-sar-config\") pod \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") "
Sep 18 16:33:54.554887 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.554869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kserve-provision-location\") pod \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\" (UID: \"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103\") "
Sep 18 16:33:54.555297 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.555170    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" (UID: "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:54.555395 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.555253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" (UID: "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:54.555395 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.555293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" (UID: "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:54.557833 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.557805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" (UID: "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:33:54.557954 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.557900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-api-access-qld4c" (OuterVolumeSpecName: "kube-api-access-qld4c") pod "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" (UID: "4a9c7bc7-e80b-4d5a-90c8-47c0f5261103"). InnerVolumeSpecName "kube-api-access-qld4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:54.656262 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.656237    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:33:54.656262 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.656264    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qld4c\" (UniqueName: \"kubernetes.io/projected/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-api-access-qld4c\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:33:54.656446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.656274    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:33:54.656446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.656283    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:33:54.656446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.656292    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:33:54.782426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.782397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:33:54.782860 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:54.782841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:33:55.099597 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.099476    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerID="fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8" exitCode=139
Sep 18 16:33:55.099597 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.099531    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"
Sep 18 16:33:55.099597 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.099579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerDied","Data":"fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8"}
Sep 18 16:33:55.099866 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.099650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j" event={"ID":"4a9c7bc7-e80b-4d5a-90c8-47c0f5261103","Type":"ContainerDied","Data":"fc3e1db9ed3d7a024b75ff90f449de2717c39a5d8aef2c78d4f8d9382de0f9dc"}
Sep 18 16:33:55.099866 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.099680    2563 scope.go:117] "RemoveContainer" containerID="c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8"
Sep 18 16:33:55.108458 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.108443    2563 scope.go:117] "RemoveContainer" containerID="fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8"
Sep 18 16:33:55.117910 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.117891    2563 scope.go:117] "RemoveContainer" containerID="911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083"
Sep 18 16:33:55.124254 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.124232    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"]
Sep 18 16:33:55.125678 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.125664    2563 scope.go:117] "RemoveContainer" containerID="c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8"
Sep 18 16:33:55.125985 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:33:55.125960    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8\": container with ID starting with c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8 not found: ID does not exist" containerID="c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8"
Sep 18 16:33:55.126086 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.125994    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8"} err="failed to get container status \"c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8\": rpc error: code = NotFound desc = could not find container \"c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8\": container with ID starting with c63472444bd9d94e1dee634caed1ab071254625148996ffcbe8143a1b98071d8 not found: ID does not exist"
Sep 18 16:33:55.126086 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.126012    2563 scope.go:117] "RemoveContainer" containerID="fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8"
Sep 18 16:33:55.126369 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:33:55.126349    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8\": container with ID starting with fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8 not found: ID does not exist" containerID="fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8"
Sep 18 16:33:55.126425 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.126376    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8"} err="failed to get container status \"fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8\": rpc error: code = NotFound desc = could not find container \"fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8\": container with ID starting with fcd80fa2b561262f349fc71e86d5a708897a99a70cbf8accf3bc72961317c7e8 not found: ID does not exist"
Sep 18 16:33:55.126425 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.126393    2563 scope.go:117] "RemoveContainer" containerID="911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083"
Sep 18 16:33:55.126662 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:33:55.126648    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083\": container with ID starting with 911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083 not found: ID does not exist" containerID="911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083"
Sep 18 16:33:55.126712 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.126667    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083"} err="failed to get container status \"911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083\": rpc error: code = NotFound desc = could not find container \"911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083\": container with ID starting with 911c51280414e94fe993ee136164c098036ae36ef6e066e48d30d6a346154083 not found: ID does not exist"
Sep 18 16:33:55.131100 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:55.131076    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-rdm8j"]
Sep 18 16:33:56.217727 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.217686    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" path="/var/lib/kubelet/pods/4a9c7bc7-e80b-4d5a-90c8-47c0f5261103/volumes"
Sep 18 16:33:56.722071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.722038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:33:56.785517 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785478    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"]
Sep 18 16:33:56.785835 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785823    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container"
Sep 18 16:33:56.785889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785838    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container"
Sep 18 16:33:56.785889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785856    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="storage-initializer"
Sep 18 16:33:56.785889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785862    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="storage-initializer"
Sep 18 16:33:56.785889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kube-rbac-proxy"
Sep 18 16:33:56.785889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785884    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kube-rbac-proxy"
Sep 18 16:33:56.786049 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785949    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kube-rbac-proxy"
Sep 18 16:33:56.786049 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.785961    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9c7bc7-e80b-4d5a-90c8-47c0f5261103" containerName="kserve-container"
Sep 18 16:33:56.789758 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.789740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.792174 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.792136    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 16:33:56.792326 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.792308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 16:33:56.807702 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.807670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"]
Sep 18 16:33:56.874425 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.874376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6fg\" (UniqueName: \"kubernetes.io/projected/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-api-access-rt6fg\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.874425 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.874426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/808cb47b-1f31-491c-9c00-64e0ad0e5565-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.874715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.874460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.874715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.874539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/808cb47b-1f31-491c-9c00-64e0ad0e5565-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.874715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.874610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.975459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.975351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/808cb47b-1f31-491c-9c00-64e0ad0e5565-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.975459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.975400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.975459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.975461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6fg\" (UniqueName: \"kubernetes.io/projected/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-api-access-rt6fg\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.975788 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.975492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/808cb47b-1f31-491c-9c00-64e0ad0e5565-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.975788 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.975531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.975906 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.975884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/808cb47b-1f31-491c-9c00-64e0ad0e5565-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.976131 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.976108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.976245 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.976226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.978225 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.978201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/808cb47b-1f31-491c-9c00-64e0ad0e5565-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:56.985843 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:56.985820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6fg\" (UniqueName: \"kubernetes.io/projected/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-api-access-rt6fg\") pod \"isvc-canary-rollback-predictor-7f7b644c9-wsgs4\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:57.100165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:57.100123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:33:57.372472 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:57.372371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"]
Sep 18 16:33:57.374773 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:33:57.374745    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808cb47b_1f31_491c_9c00_64e0ad0e5565.slice/crio-8b16770b2fac42833f40d6a56025f0edc8df4652cf6bcd4fe67fe960c60ae776 WatchSource:0}: Error finding container 8b16770b2fac42833f40d6a56025f0edc8df4652cf6bcd4fe67fe960c60ae776: Status 404 returned error can't find the container with id 8b16770b2fac42833f40d6a56025f0edc8df4652cf6bcd4fe67fe960c60ae776
Sep 18 16:33:58.114250 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:58.114162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerStarted","Data":"608b69f8348b512433ad58ef4866a003deb93531202b1f376504eb89b4781c8e"}
Sep 18 16:33:58.114250 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:33:58.114219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerStarted","Data":"8b16770b2fac42833f40d6a56025f0edc8df4652cf6bcd4fe67fe960c60ae776"}
Sep 18 16:34:02.035808 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.035773    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"]
Sep 18 16:34:02.036179 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.036125    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" containerID="cri-o://a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237" gracePeriod=30
Sep 18 16:34:02.036179 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.036147    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" containerID="cri-o://e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918" gracePeriod=30
Sep 18 16:34:02.036266 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.036163    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" containerID="cri-o://6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0" gracePeriod=30
Sep 18 16:34:02.050487 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.050458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:02.132099 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.132069    2563 generic.go:356] "Generic (PLEG): container finished" podID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerID="608b69f8348b512433ad58ef4866a003deb93531202b1f376504eb89b4781c8e" exitCode=0
Sep 18 16:34:02.132275 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.132146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerDied","Data":"608b69f8348b512433ad58ef4866a003deb93531202b1f376504eb89b4781c8e"}
Sep 18 16:34:02.164359 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.164331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:02.348230 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.348134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:02.641025 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.640996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:02.820778 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:02.820741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:03.138763 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:03.138730    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerID="6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0" exitCode=2
Sep 18 16:34:03.139203 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:03.138804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerDied","Data":"6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0"}
Sep 18 16:34:03.140913 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:03.140890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerStarted","Data":"2c98c8fd614f365d9ad0c5f78d1560ec8bda6b959d006cceccedac90b3dd1eeb"}
Sep 18 16:34:03.141022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:03.140949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerStarted","Data":"4a14326a6cc1686f36c3a2bccc3502131957f525a9891f14388af19de58c79fa"}
Sep 18 16:34:03.141166 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:03.141149    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:34:03.162577 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:03.162505    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podStartSLOduration=3.328958118 podStartE2EDuration="7.162486554s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833528436s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:03.160601804 +0000 UTC m=+1055.160862640" watchObservedRunningTime="2026-09-18 16:34:03.162486554 +0000 UTC m=+1055.162747365"
Sep 18 16:34:04.145219 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:04.145183    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:34:04.146510 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:04.146482    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:34:04.776931 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:04.776891    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:34:04.781689 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:04.781659    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:34:04.783888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:04.783854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:34:05.149631 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:05.149593    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:34:07.161242 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:07.161214    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerID="a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237" exitCode=0
Sep 18 16:34:07.161616 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:07.161263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerDied","Data":"a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237"}
Sep 18 16:34:09.776923 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:09.776838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:34:10.153905 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:10.153878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:34:10.154501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:10.154472    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:34:14.776281 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:14.776230    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:34:14.776733 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:14.776382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:34:14.781468 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:14.781431    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:34:14.783697 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:14.783673    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:34:19.776479 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:19.776432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:34:20.154804 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:20.154766    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:34:23.628533 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:23.628494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:34:24.776574 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:24.776529    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:34:24.781244 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:24.781211    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:34:24.781375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:24.781356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:34:24.783623 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:24.783600    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 18 16:34:24.783734 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:24.783716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:34:29.776870 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:29.776817    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 18 16:34:30.154580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:30.154526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:34:32.192399 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.192375    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:34:32.260976 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.260948    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerID="e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918" exitCode=0
Sep 18 16:34:32.261138 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.261008    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"
Sep 18 16:34:32.261138 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.261014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerDied","Data":"e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918"}
Sep 18 16:34:32.261138 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.261074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4" event={"ID":"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5","Type":"ContainerDied","Data":"6f431dd860a42598df70f4c16d7d45670be9fc80f248f8a7f3bb237a0a92dcb5"}
Sep 18 16:34:32.261138 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.261104    2563 scope.go:117] "RemoveContainer" containerID="e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918"
Sep 18 16:34:32.269006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.268988    2563 scope.go:117] "RemoveContainer" containerID="6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0"
Sep 18 16:34:32.276237 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.276220    2563 scope.go:117] "RemoveContainer" containerID="a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237"
Sep 18 16:34:32.283445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.283426    2563 scope.go:117] "RemoveContainer" containerID="95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4"
Sep 18 16:34:32.291070 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291052    2563 scope.go:117] "RemoveContainer" containerID="e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918"
Sep 18 16:34:32.291357 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:34:32.291339    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918\": container with ID starting with e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918 not found: ID does not exist" containerID="e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918"
Sep 18 16:34:32.291404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291367    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918"} err="failed to get container status \"e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918\": rpc error: code = NotFound desc = could not find container \"e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918\": container with ID starting with e5446cabd06be67db35cf767881ffdda6405111042f499ce522d36f92c1ff918 not found: ID does not exist"
Sep 18 16:34:32.291404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291387    2563 scope.go:117] "RemoveContainer" containerID="6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0"
Sep 18 16:34:32.291669 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:34:32.291651    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0\": container with ID starting with 6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0 not found: ID does not exist" containerID="6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0"
Sep 18 16:34:32.291731 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291674    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0"} err="failed to get container status \"6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0\": rpc error: code = NotFound desc = could not find container \"6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0\": container with ID starting with 6039a7c85d2e0274fe54f8dc98b6ad56985901a54256eeb6fc5a8a3834c746c0 not found: ID does not exist"
Sep 18 16:34:32.291731 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291689    2563 scope.go:117] "RemoveContainer" containerID="a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237"
Sep 18 16:34:32.291949 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:34:32.291930    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237\": container with ID starting with a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237 not found: ID does not exist" containerID="a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237"
Sep 18 16:34:32.292022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291959    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237"} err="failed to get container status \"a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237\": rpc error: code = NotFound desc = could not find container \"a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237\": container with ID starting with a119d0a5a2c00ecbe933aa954003ce8bceb6882bb6d2d91d21e2df69545ec237 not found: ID does not exist"
Sep 18 16:34:32.292022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.291981    2563 scope.go:117] "RemoveContainer" containerID="95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4"
Sep 18 16:34:32.292255 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:34:32.292230    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4\": container with ID starting with 95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4 not found: ID does not exist" containerID="95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4"
Sep 18 16:34:32.292304 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.292251    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4"} err="failed to get container status \"95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4\": rpc error: code = NotFound desc = could not find container \"95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4\": container with ID starting with 95638703ef66d5806e4de53e9cb91f86a7640ec1ca5797f92f7fafd542ac57e4 not found: ID does not exist"
Sep 18 16:34:32.389913 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.389881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-rbac-proxy-sar-config\") pod \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") "
Sep 18 16:34:32.389913 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.389919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ws68\" (UniqueName: \"kubernetes.io/projected/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-api-access-7ws68\") pod \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") "
Sep 18 16:34:32.390115 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.389943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-proxy-tls\") pod \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") "
Sep 18 16:34:32.390115 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.390047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-cabundle-cert\") pod \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") "
Sep 18 16:34:32.390115 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.390099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kserve-provision-location\") pod \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\" (UID: \"c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5\") "
Sep 18 16:34:32.390696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.390356    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" (UID: "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:32.390696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.390418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" (UID: "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:32.390696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.390428    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:34:32.390696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.390436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" (UID: "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:32.392214 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.392184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-api-access-7ws68" (OuterVolumeSpecName: "kube-api-access-7ws68") pod "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" (UID: "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5"). InnerVolumeSpecName "kube-api-access-7ws68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:32.392288 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.392271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" (UID: "c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:32.491744 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.491707    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:34:32.491744 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.491739    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ws68\" (UniqueName: \"kubernetes.io/projected/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-kube-api-access-7ws68\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:34:32.491955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.491752    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:34:32.491955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.491766    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:34:32.583624 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.583595    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"]
Sep 18 16:34:32.587371 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:32.587343    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-3f0a6-predictor-5c676957f8-tf5m4"]
Sep 18 16:34:34.217643 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:34.217612    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" path="/var/lib/kubelet/pods/c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5/volumes"
Sep 18 16:34:40.155018 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:40.154978    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:34:50.154598 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:34:50.154540    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:35:00.154992 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:00.154947    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:35:10.154827 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:10.154784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:35:20.155479 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:20.155446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:35:26.748717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:26.748651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"]
Sep 18 16:35:26.749172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:26.748917    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" containerID="cri-o://4a14326a6cc1686f36c3a2bccc3502131957f525a9891f14388af19de58c79fa" gracePeriod=30
Sep 18 16:35:26.749172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:26.748945    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kube-rbac-proxy" containerID="cri-o://2c98c8fd614f365d9ad0c5f78d1560ec8bda6b959d006cceccedac90b3dd1eeb" gracePeriod=30
Sep 18 16:35:26.773696 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:26.773669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:27.487790 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:27.487749    2563 generic.go:356] "Generic (PLEG): container finished" podID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerID="2c98c8fd614f365d9ad0c5f78d1560ec8bda6b959d006cceccedac90b3dd1eeb" exitCode=2
Sep 18 16:35:27.487975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:27.487818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerDied","Data":"2c98c8fd614f365d9ad0c5f78d1560ec8bda6b959d006cceccedac90b3dd1eeb"}
Sep 18 16:35:30.150066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:30.150017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Sep 18 16:35:30.155374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:30.155353    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 18 16:35:31.506659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.506622    2563 generic.go:356] "Generic (PLEG): container finished" podID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerID="4a14326a6cc1686f36c3a2bccc3502131957f525a9891f14388af19de58c79fa" exitCode=0
Sep 18 16:35:31.506659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.506664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerDied","Data":"4a14326a6cc1686f36c3a2bccc3502131957f525a9891f14388af19de58c79fa"}
Sep 18 16:35:31.601458 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.601432    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:35:31.775270 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-rbac-proxy-sar-config\") pod \"808cb47b-1f31-491c-9c00-64e0ad0e5565\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") "
Sep 18 16:35:31.775270 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/808cb47b-1f31-491c-9c00-64e0ad0e5565-kserve-provision-location\") pod \"808cb47b-1f31-491c-9c00-64e0ad0e5565\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") "
Sep 18 16:35:31.775500 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/808cb47b-1f31-491c-9c00-64e0ad0e5565-proxy-tls\") pod \"808cb47b-1f31-491c-9c00-64e0ad0e5565\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") "
Sep 18 16:35:31.775500 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt6fg\" (UniqueName: \"kubernetes.io/projected/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-api-access-rt6fg\") pod \"808cb47b-1f31-491c-9c00-64e0ad0e5565\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") "
Sep 18 16:35:31.775500 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-cabundle-cert\") pod \"808cb47b-1f31-491c-9c00-64e0ad0e5565\" (UID: \"808cb47b-1f31-491c-9c00-64e0ad0e5565\") "
Sep 18 16:35:31.775684 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "808cb47b-1f31-491c-9c00-64e0ad0e5565" (UID: "808cb47b-1f31-491c-9c00-64e0ad0e5565"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:35:31.775728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808cb47b-1f31-491c-9c00-64e0ad0e5565-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "808cb47b-1f31-491c-9c00-64e0ad0e5565" (UID: "808cb47b-1f31-491c-9c00-64e0ad0e5565"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:31.775833 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.775792    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "808cb47b-1f31-491c-9c00-64e0ad0e5565" (UID: "808cb47b-1f31-491c-9c00-64e0ad0e5565"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:35:31.777490 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.777472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808cb47b-1f31-491c-9c00-64e0ad0e5565-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "808cb47b-1f31-491c-9c00-64e0ad0e5565" (UID: "808cb47b-1f31-491c-9c00-64e0ad0e5565"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:35:31.777643 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.777628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-api-access-rt6fg" (OuterVolumeSpecName: "kube-api-access-rt6fg") pod "808cb47b-1f31-491c-9c00-64e0ad0e5565" (UID: "808cb47b-1f31-491c-9c00-64e0ad0e5565"). InnerVolumeSpecName "kube-api-access-rt6fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:31.876776 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.876746    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.876776 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.876774    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/808cb47b-1f31-491c-9c00-64e0ad0e5565-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.876955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.876785    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/808cb47b-1f31-491c-9c00-64e0ad0e5565-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.876955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.876797    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt6fg\" (UniqueName: \"kubernetes.io/projected/808cb47b-1f31-491c-9c00-64e0ad0e5565-kube-api-access-rt6fg\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.876955 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.876806    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/808cb47b-1f31-491c-9c00-64e0ad0e5565-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:35:31.901050 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.901016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:31.955232 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955194    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"]
Sep 18 16:35:31.955613 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955594    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955615    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955626    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955646    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kube-rbac-proxy"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kube-rbac-proxy"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955666    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="storage-initializer"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="storage-initializer"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955684    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container"
Sep 18 16:35:31.955709 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955708    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="storage-initializer"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="storage-initializer"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955735    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955821    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kube-rbac-proxy"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955831    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kube-rbac-proxy"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955840    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="kserve-container"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955849    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec894f-26e8-4f3f-a3c7-339bb7bb07d5" containerName="agent"
Sep 18 16:35:31.956186 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.955859    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" containerName="kserve-container"
Sep 18 16:35:31.959304 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.959285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:31.961544 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.961521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 16:35:31.961656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.961634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 16:35:31.965802 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.965780    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"]
Sep 18 16:35:31.969160 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.969140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:31.969914 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.969896    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"]
Sep 18 16:35:31.971188 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.971171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 16:35:31.974369 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.974347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:31.982200 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:31.982171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"]
Sep 18 16:35:32.078258 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c657f83-3411-489e-a770-b0713ed17508-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.078258 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c657f83-3411-489e-a770-b0713ed17508-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.078258 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjq2v\" (UniqueName: \"kubernetes.io/projected/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-api-access-xjq2v\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlm44\" (UniqueName: \"kubernetes.io/projected/3c657f83-3411-489e-a770-b0713ed17508-kube-api-access-hlm44\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.078501 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.078477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.128852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.128818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:32.179641 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.179641 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjq2v\" (UniqueName: \"kubernetes.io/projected/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-api-access-xjq2v\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.179852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.179852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.179852 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlm44\" (UniqueName: \"kubernetes.io/projected/3c657f83-3411-489e-a770-b0713ed17508-kube-api-access-hlm44\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.179996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.180367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.180367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c657f83-3411-489e-a770-b0713ed17508-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c657f83-3411-489e-a770-b0713ed17508-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.180708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c657f83-3411-489e-a770-b0713ed17508-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.180708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.180708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.180695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.182768 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.182745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.182768 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.182759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c657f83-3411-489e-a770-b0713ed17508-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.187889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.187862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjq2v\" (UniqueName: \"kubernetes.io/projected/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-api-access-xjq2v\") pod \"isvc-canary-stop-predictor-5ffdccb94c-zpw8r\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.187889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.187872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlm44\" (UniqueName: \"kubernetes.io/projected/3c657f83-3411-489e-a770-b0713ed17508-kube-api-access-hlm44\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.229089 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.229064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:32.270129 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.270101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:32.281338 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.281316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:32.513000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.512973    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"
Sep 18 16:35:32.513000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.512990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4" event={"ID":"808cb47b-1f31-491c-9c00-64e0ad0e5565","Type":"ContainerDied","Data":"8b16770b2fac42833f40d6a56025f0edc8df4652cf6bcd4fe67fe960c60ae776"}
Sep 18 16:35:32.513485 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.513063    2563 scope.go:117] "RemoveContainer" containerID="2c98c8fd614f365d9ad0c5f78d1560ec8bda6b959d006cceccedac90b3dd1eeb"
Sep 18 16:35:32.521400 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.521378    2563 scope.go:117] "RemoveContainer" containerID="4a14326a6cc1686f36c3a2bccc3502131957f525a9891f14388af19de58c79fa"
Sep 18 16:35:32.530044 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.529947    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"]
Sep 18 16:35:32.530107 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.530092    2563 scope.go:117] "RemoveContainer" containerID="608b69f8348b512433ad58ef4866a003deb93531202b1f376504eb89b4781c8e"
Sep 18 16:35:32.534699 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.534677    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-wsgs4"]
Sep 18 16:35:32.548514 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.548492    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"]
Sep 18 16:35:32.551615 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:35:32.551591    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c657f83_3411_489e_a770_b0713ed17508.slice/crio-33fe476106da134557844471c2dd726bb9d9e09dc11074b88956daaf5f85bd5f WatchSource:0}: Error finding container 33fe476106da134557844471c2dd726bb9d9e09dc11074b88956daaf5f85bd5f: Status 404 returned error can't find the container with id 33fe476106da134557844471c2dd726bb9d9e09dc11074b88956daaf5f85bd5f
Sep 18 16:35:32.563942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:32.563925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"]
Sep 18 16:35:32.567867 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:35:32.567842    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f41ccb_5c02_437e_b3c5_c93205cf3c7d.slice/crio-0bbd05f0b1f5b278d8e01159ce57437f5c743acd2ae59af5389b17571d78f492 WatchSource:0}: Error finding container 0bbd05f0b1f5b278d8e01159ce57437f5c743acd2ae59af5389b17571d78f492: Status 404 returned error can't find the container with id 0bbd05f0b1f5b278d8e01159ce57437f5c743acd2ae59af5389b17571d78f492
Sep 18 16:35:33.518950 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:33.518908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerStarted","Data":"794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7"}
Sep 18 16:35:33.519419 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:33.519026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerStarted","Data":"0bbd05f0b1f5b278d8e01159ce57437f5c743acd2ae59af5389b17571d78f492"}
Sep 18 16:35:33.520611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:33.520589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerStarted","Data":"a21d0dc4f320dfa6685964287202bcc088fda6bea69bef5b2fc7a0a3c28566d1"}
Sep 18 16:35:33.520716 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:33.520646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerStarted","Data":"33fe476106da134557844471c2dd726bb9d9e09dc11074b88956daaf5f85bd5f"}
Sep 18 16:35:34.217818 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:34.217787    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808cb47b-1f31-491c-9c00-64e0ad0e5565" path="/var/lib/kubelet/pods/808cb47b-1f31-491c-9c00-64e0ad0e5565/volumes"
Sep 18 16:35:36.536073 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:36.536035    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerID="794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7" exitCode=0
Sep 18 16:35:36.536506 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:36.536107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerDied","Data":"794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7"}
Sep 18 16:35:36.537744 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:36.537725    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c657f83-3411-489e-a770-b0713ed17508" containerID="a21d0dc4f320dfa6685964287202bcc088fda6bea69bef5b2fc7a0a3c28566d1" exitCode=0
Sep 18 16:35:36.537835 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:36.537753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerDied","Data":"a21d0dc4f320dfa6685964287202bcc088fda6bea69bef5b2fc7a0a3c28566d1"}
Sep 18 16:35:37.544440 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.544406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerStarted","Data":"bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a"}
Sep 18 16:35:37.544918 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.544506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerStarted","Data":"4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922"}
Sep 18 16:35:37.544918 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.544821    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:37.546594 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.546552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerStarted","Data":"f77464021bd3066bf523ebdf73881220f851d14f315897fcb2e1e5632555e7ee"}
Sep 18 16:35:37.546692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.546617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerStarted","Data":"73b1a98e3eb2ed83bf36af472e5adf2022431455d9c357cba9d67096e120ae10"}
Sep 18 16:35:37.546873 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.546856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:37.546912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.546902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:37.548407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.548386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:35:37.563060 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.563023    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podStartSLOduration=2.792201246 podStartE2EDuration="6.563012453s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770811207s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:37.561958343 +0000 UTC m=+1149.562219170" watchObservedRunningTime="2026-09-18 16:35:37.563012453 +0000 UTC m=+1149.563273263"
Sep 18 16:35:37.581132 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:37.581084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podStartSLOduration=2.887703392 podStartE2EDuration="6.581066833s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.693363441s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:37.579672218 +0000 UTC m=+1149.579933056" watchObservedRunningTime="2026-09-18 16:35:37.581066833 +0000 UTC m=+1149.581327644"
Sep 18 16:35:38.550624 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:38.550585    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:35:38.551044 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:38.550760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:38.551992 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:38.551966    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:35:39.554780 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:39.554742    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:35:43.555824 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:43.555794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:35:43.556323 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:43.556298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:35:44.559078 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:44.559048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:35:44.559635 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:44.559610    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:35:52.230054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:52.229983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:52.744309 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:52.744275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:35:53.556238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:53.556201    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:35:54.559500 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:35:54.559462    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:02.296521 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.296453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:36:02.513991 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.513951    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"]
Sep 18 16:36:02.521253 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.521224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.523616 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.523593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-serving-cert\""
Sep 18 16:36:02.523745 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.523596    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-c8b7d-kube-rbac-proxy-sar-config\""
Sep 18 16:36:02.528977 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.528893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"]
Sep 18 16:36:02.605855 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.605776    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"]
Sep 18 16:36:02.609371 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.609354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.612041 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.612017    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c8b7d-kube-rbac-proxy-sar-config\""
Sep 18 16:36:02.612193 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.612019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-serving-cert\""
Sep 18 16:36:02.620162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.620138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"]
Sep 18 16:36:02.633692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.633659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.633692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.633697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.633937 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.633731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh6z\" (UniqueName: \"kubernetes.io/projected/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-api-access-bxh6z\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.633937 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.633756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.633937 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.633830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tnd\" (UniqueName: \"kubernetes.io/projected/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-api-access-t2tnd\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh6z\" (UniqueName: \"kubernetes.io/projected/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-api-access-bxh6z\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:36:02.735237    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-c8b7d-predictor-serving-cert" not found
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:36:02.735318    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls podName:5ebe377b-af8c-4bbf-8641-74af533ca5d5 nodeName:}" failed. No retries permitted until 2026-09-18 16:36:03.235301488 +0000 UTC m=+1175.235562281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" (UID: "5ebe377b-af8c-4bbf-8641-74af533ca5d5") : secret "isvc-sklearn-graph-raw-hpa-c8b7d-predictor-serving-cert" not found
Sep 18 16:36:02.735393 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.735849 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.735962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.736001 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.735940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.743454 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.743424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh6z\" (UniqueName: \"kubernetes.io/projected/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-api-access-bxh6z\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:02.836450 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.836405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.836450 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.836450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.836729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.836606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.836729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.836679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.836729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.836701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tnd\" (UniqueName: \"kubernetes.io/projected/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-api-access-t2tnd\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.836849 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:36:02.836791    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-c8b7d-predictor-serving-cert" not found
Sep 18 16:36:02.836884 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:36:02.836873    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls podName:d569322a-afa8-40df-b5ba-fd4b9b8869bd nodeName:}" failed. No retries permitted until 2026-09-18 16:36:03.336851923 +0000 UTC m=+1175.337112715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" (UID: "d569322a-afa8-40df-b5ba-fd4b9b8869bd") : secret "isvc-xgboost-graph-raw-hpa-c8b7d-predictor-serving-cert" not found
Sep 18 16:36:02.837000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.836977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.837192 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.837173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.837237 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.837218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:02.845372 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:02.845349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tnd\" (UniqueName: \"kubernetes.io/projected/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-api-access-t2tnd\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:03.240801 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.240760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:03.243340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.243307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:03.341868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.341824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:03.344425 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.344400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:03.433486 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.433445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:03.522546 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.522462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:03.557160 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.557121    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:36:03.710470 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.710445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"]
Sep 18 16:36:03.712457 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:36:03.712431    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebe377b_af8c_4bbf_8641_74af533ca5d5.slice/crio-58224a1b0fe74f07de04b69fe150998d4f41d3c950874bc7d31d6bc845eb06af WatchSource:0}: Error finding container 58224a1b0fe74f07de04b69fe150998d4f41d3c950874bc7d31d6bc845eb06af: Status 404 returned error can't find the container with id 58224a1b0fe74f07de04b69fe150998d4f41d3c950874bc7d31d6bc845eb06af
Sep 18 16:36:03.781309 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.781284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:36:03.795915 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:03.795855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"]
Sep 18 16:36:03.798174 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:36:03.798150    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd569322a_afa8_40df_b5ba_fd4b9b8869bd.slice/crio-2fbed06e1ccfbda9a40a69dc0095e1adf61139d50a0788b7f00410ea63e513b4 WatchSource:0}: Error finding container 2fbed06e1ccfbda9a40a69dc0095e1adf61139d50a0788b7f00410ea63e513b4: Status 404 returned error can't find the container with id 2fbed06e1ccfbda9a40a69dc0095e1adf61139d50a0788b7f00410ea63e513b4
Sep 18 16:36:04.560158 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:04.560112    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:04.657281 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:04.657239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerStarted","Data":"b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3"}
Sep 18 16:36:04.657491 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:04.657313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerStarted","Data":"2fbed06e1ccfbda9a40a69dc0095e1adf61139d50a0788b7f00410ea63e513b4"}
Sep 18 16:36:04.658858 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:04.658831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerStarted","Data":"601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c"}
Sep 18 16:36:04.658964 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:04.658881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerStarted","Data":"58224a1b0fe74f07de04b69fe150998d4f41d3c950874bc7d31d6bc845eb06af"}
Sep 18 16:36:07.673919 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:07.673882    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerID="601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c" exitCode=0
Sep 18 16:36:07.673919 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:07.673923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerDied","Data":"601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c"}
Sep 18 16:36:08.680289 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.680247    2563 generic.go:356] "Generic (PLEG): container finished" podID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerID="b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3" exitCode=0
Sep 18 16:36:08.680766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.680320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerDied","Data":"b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3"}
Sep 18 16:36:08.682034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.682010    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:36:08.683574 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.683530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerStarted","Data":"bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97"}
Sep 18 16:36:08.683664 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.683625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerStarted","Data":"07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e"}
Sep 18 16:36:08.683976 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.683958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:08.684061 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.684015    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:08.685356 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.685334    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:08.716194 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:08.716148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podStartSLOduration=3.179868019 podStartE2EDuration="6.716133909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.53626589s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:36:08.714647942 +0000 UTC m=+1180.714908773" watchObservedRunningTime="2026-09-18 16:36:08.716133909 +0000 UTC m=+1180.716394719"
Sep 18 16:36:09.691068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:09.691021    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:13.556649 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:13.556607    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:36:14.559581 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:14.559516    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:14.696866 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:14.696836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:36:14.697447 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:14.697417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:22.506470 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:22.506421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:36:23.556739 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:23.556700    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:36:24.559532 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:24.559485    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:24.698228 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:24.698182    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:28.792907 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:28.792870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerStarted","Data":"31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa"}
Sep 18 16:36:28.793274 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:28.792963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerStarted","Data":"1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990"}
Sep 18 16:36:28.793274 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:28.793158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:28.812666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:28.812618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podStartSLOduration=3.256495434 podStartE2EDuration="26.812603713s" totalImagesPullingTime="19.493146133s" totalInitContainerRuntime="4.062962146s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:36:28.810737514 +0000 UTC m=+1200.810998328" watchObservedRunningTime="2026-09-18 16:36:28.812603713 +0000 UTC m=+1200.812864521"
Sep 18 16:36:29.797603 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:29.797547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:29.798995 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:29.798968    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:36:30.157460 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:30.157435    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:36:30.157660 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:30.157484    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:36:30.802443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:30.802406    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:36:33.557024 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:33.556982    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:36:34.559791 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:34.559753    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:34.698039 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:34.698004    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:35.807946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:35.807917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:36:35.808431 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:35.808400    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:36:43.556757 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:43.556709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:36:44.560189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:44.560134    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:44.697952 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:44.697906    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:45.808748 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:45.808702    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:36:46.213122 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:46.213080    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:49.212719 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:49.212678    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:36:54.698289 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:54.698248    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:36:55.808371 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:55.808334    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:36:56.213280 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:56.213234    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 16:36:59.212864 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:36:59.212823    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 18 16:37:04.697859 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:04.697819    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:37:05.808359 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:05.808323    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:37:06.218440 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:06.218413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:37:09.213182 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.213101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:37:09.339813 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.339780    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"]
Sep 18 16:37:09.340030 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.340003    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container" containerID="cri-o://4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922" gracePeriod=30
Sep 18 16:37:09.340133 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.340034    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kube-rbac-proxy" containerID="cri-o://bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a" gracePeriod=30
Sep 18 16:37:09.555079 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.555032    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 16:37:09.975982 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.975949    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerID="bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a" exitCode=2
Sep 18 16:37:09.976151 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:09.976005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerDied","Data":"bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a"}
Sep 18 16:37:11.952707 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:11.952670    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"]
Sep 18 16:37:11.953082 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:11.952892    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container" containerID="cri-o://73b1a98e3eb2ed83bf36af472e5adf2022431455d9c357cba9d67096e120ae10" gracePeriod=30
Sep 18 16:37:11.953082 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:11.952976    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kube-rbac-proxy" containerID="cri-o://f77464021bd3066bf523ebdf73881220f851d14f315897fcb2e1e5632555e7ee" gracePeriod=30
Sep 18 16:37:11.970193 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:11.970169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:12.034098 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:12.034065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:12.994662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:12.994628    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c657f83-3411-489e-a770-b0713ed17508" containerID="f77464021bd3066bf523ebdf73881220f851d14f315897fcb2e1e5632555e7ee" exitCode=2
Sep 18 16:37:12.995048 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:12.994703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerDied","Data":"f77464021bd3066bf523ebdf73881220f851d14f315897fcb2e1e5632555e7ee"}
Sep 18 16:37:13.551014 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:13.550974    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 18 16:37:14.555774 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.555729    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 16:37:14.697848 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.697814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:37:14.891834 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.891812    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:37:14.967761 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.967730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kserve-provision-location\") pod \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") "
Sep 18 16:37:14.967968 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.967781    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-rbac-proxy-sar-config\") pod \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") "
Sep 18 16:37:14.967968 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.967804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjq2v\" (UniqueName: \"kubernetes.io/projected/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-api-access-xjq2v\") pod \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") "
Sep 18 16:37:14.967968 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.967895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-cabundle-cert\") pod \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") "
Sep 18 16:37:14.967968 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.967924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-proxy-tls\") pod \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\" (UID: \"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d\") "
Sep 18 16:37:14.968205 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.968171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" (UID: "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:14.968263 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.968137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" (UID: "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:14.968301 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.968274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" (UID: "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:14.970043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.970022    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-api-access-xjq2v" (OuterVolumeSpecName: "kube-api-access-xjq2v") pod "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" (UID: "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d"). InnerVolumeSpecName "kube-api-access-xjq2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:14.970130 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:14.970110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" (UID: "a4f41ccb-5c02-437e-b3c5-c93205cf3c7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:37:15.005293 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.005262    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerID="4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922" exitCode=0
Sep 18 16:37:15.005488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.005332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"
Sep 18 16:37:15.005488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.005346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerDied","Data":"4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922"}
Sep 18 16:37:15.005488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.005407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r" event={"ID":"a4f41ccb-5c02-437e-b3c5-c93205cf3c7d","Type":"ContainerDied","Data":"0bbd05f0b1f5b278d8e01159ce57437f5c743acd2ae59af5389b17571d78f492"}
Sep 18 16:37:15.005488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.005435    2563 scope.go:117] "RemoveContainer" containerID="bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a"
Sep 18 16:37:15.014667 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.014640    2563 scope.go:117] "RemoveContainer" containerID="4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922"
Sep 18 16:37:15.024483 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.024465    2563 scope.go:117] "RemoveContainer" containerID="794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7"
Sep 18 16:37:15.027520 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.027496    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"]
Sep 18 16:37:15.033272 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.033249    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-zpw8r"]
Sep 18 16:37:15.034465 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.034451    2563 scope.go:117] "RemoveContainer" containerID="bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a"
Sep 18 16:37:15.034805 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:37:15.034781    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a\": container with ID starting with bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a not found: ID does not exist" containerID="bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a"
Sep 18 16:37:15.034875 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.034816    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a"} err="failed to get container status \"bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a\": rpc error: code = NotFound desc = could not find container \"bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a\": container with ID starting with bbfc76eb621744f58269cf005cab330295cbcf435c05fda1b5937b6e4be9591a not found: ID does not exist"
Sep 18 16:37:15.034875 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.034843    2563 scope.go:117] "RemoveContainer" containerID="4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922"
Sep 18 16:37:15.035093 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:37:15.035076    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922\": container with ID starting with 4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922 not found: ID does not exist" containerID="4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922"
Sep 18 16:37:15.035139 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.035098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922"} err="failed to get container status \"4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922\": rpc error: code = NotFound desc = could not find container \"4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922\": container with ID starting with 4b0405719c453763907772902b27e12be46d0db5f1d4a728364b568407049922 not found: ID does not exist"
Sep 18 16:37:15.035139 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.035110    2563 scope.go:117] "RemoveContainer" containerID="794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7"
Sep 18 16:37:15.035325 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:37:15.035307    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7\": container with ID starting with 794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7 not found: ID does not exist" containerID="794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7"
Sep 18 16:37:15.035362 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.035330    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7"} err="failed to get container status \"794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7\": rpc error: code = NotFound desc = could not find container \"794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7\": container with ID starting with 794cd72e6a688b037b51d57a9814cbfc846f82608dfd2970c6910f1096d2e0f7 not found: ID does not exist"
Sep 18 16:37:15.069001 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.068931    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:15.069001 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.068958    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:15.069001 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.068967    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:15.069001 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.068976    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:15.069001 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.068985    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjq2v\" (UniqueName: \"kubernetes.io/projected/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d-kube-api-access-xjq2v\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:15.808429 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:15.808393    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:37:16.217436 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:16.217402    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" path="/var/lib/kubelet/pods/a4f41ccb-5c02-437e-b3c5-c93205cf3c7d/volumes"
Sep 18 16:37:17.018316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.018282    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c657f83-3411-489e-a770-b0713ed17508" containerID="73b1a98e3eb2ed83bf36af472e5adf2022431455d9c357cba9d67096e120ae10" exitCode=0
Sep 18 16:37:17.018720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.018325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerDied","Data":"73b1a98e3eb2ed83bf36af472e5adf2022431455d9c357cba9d67096e120ae10"}
Sep 18 16:37:17.099603 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.099583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:37:17.186836 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.186796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c657f83-3411-489e-a770-b0713ed17508-kserve-provision-location\") pod \"3c657f83-3411-489e-a770-b0713ed17508\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") "
Sep 18 16:37:17.187028 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.186846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-cabundle-cert\") pod \"3c657f83-3411-489e-a770-b0713ed17508\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") "
Sep 18 16:37:17.187028 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.186872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-kube-rbac-proxy-sar-config\") pod \"3c657f83-3411-489e-a770-b0713ed17508\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") "
Sep 18 16:37:17.187028 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.186944    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlm44\" (UniqueName: \"kubernetes.io/projected/3c657f83-3411-489e-a770-b0713ed17508-kube-api-access-hlm44\") pod \"3c657f83-3411-489e-a770-b0713ed17508\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") "
Sep 18 16:37:17.187028 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.186990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c657f83-3411-489e-a770-b0713ed17508-proxy-tls\") pod \"3c657f83-3411-489e-a770-b0713ed17508\" (UID: \"3c657f83-3411-489e-a770-b0713ed17508\") "
Sep 18 16:37:17.187275 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.187189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c657f83-3411-489e-a770-b0713ed17508-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c657f83-3411-489e-a770-b0713ed17508" (UID: "3c657f83-3411-489e-a770-b0713ed17508"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:17.187275 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.187224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c657f83-3411-489e-a770-b0713ed17508" (UID: "3c657f83-3411-489e-a770-b0713ed17508"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:17.187394 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.187273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c657f83-3411-489e-a770-b0713ed17508" (UID: "3c657f83-3411-489e-a770-b0713ed17508"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:37:17.189167 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.189142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c657f83-3411-489e-a770-b0713ed17508-kube-api-access-hlm44" (OuterVolumeSpecName: "kube-api-access-hlm44") pod "3c657f83-3411-489e-a770-b0713ed17508" (UID: "3c657f83-3411-489e-a770-b0713ed17508"). InnerVolumeSpecName "kube-api-access-hlm44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:17.189311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.189237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c657f83-3411-489e-a770-b0713ed17508-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c657f83-3411-489e-a770-b0713ed17508" (UID: "3c657f83-3411-489e-a770-b0713ed17508"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:37:17.288165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.288121    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:17.288165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.288147    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c657f83-3411-489e-a770-b0713ed17508-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:17.288165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.288157    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlm44\" (UniqueName: \"kubernetes.io/projected/3c657f83-3411-489e-a770-b0713ed17508-kube-api-access-hlm44\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:17.288165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.288165    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c657f83-3411-489e-a770-b0713ed17508-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:17.288165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:17.288174    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c657f83-3411-489e-a770-b0713ed17508-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:37:18.024818 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.024788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7" event={"ID":"3c657f83-3411-489e-a770-b0713ed17508","Type":"ContainerDied","Data":"33fe476106da134557844471c2dd726bb9d9e09dc11074b88956daaf5f85bd5f"}
Sep 18 16:37:18.024818 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.024818    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"
Sep 18 16:37:18.025381 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.024853    2563 scope.go:117] "RemoveContainer" containerID="f77464021bd3066bf523ebdf73881220f851d14f315897fcb2e1e5632555e7ee"
Sep 18 16:37:18.033371 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.033354    2563 scope.go:117] "RemoveContainer" containerID="73b1a98e3eb2ed83bf36af472e5adf2022431455d9c357cba9d67096e120ae10"
Sep 18 16:37:18.040950 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.040934    2563 scope.go:117] "RemoveContainer" containerID="a21d0dc4f320dfa6685964287202bcc088fda6bea69bef5b2fc7a0a3c28566d1"
Sep 18 16:37:18.046458 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.046437    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"]
Sep 18 16:37:18.050404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.050384    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-p7bh7"]
Sep 18 16:37:18.223050 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:18.223017    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c657f83-3411-489e-a770-b0713ed17508" path="/var/lib/kubelet/pods/3c657f83-3411-489e-a770-b0713ed17508/volumes"
Sep 18 16:37:22.221515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:22.221486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:22.940223 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:22.940192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:24.697993 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:24.697959    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:37:25.808809 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:25.808770    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:37:34.698577 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:34.698540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:37:35.809497 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:35.809464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:37:52.558285 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:52.558246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:37:52.732600 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:37:52.732523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:02.620965 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.620927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:02.699418 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.699372    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"]
Sep 18 16:38:02.699716 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.699691    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" containerID="cri-o://07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e" gracePeriod=30
Sep 18 16:38:02.699813 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.699755    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kube-rbac-proxy" containerID="cri-o://bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97" gracePeriod=30
Sep 18 16:38:02.724369 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.724328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:02.738137 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.738102    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"]
Sep 18 16:38:02.738402 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.738355    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" containerID="cri-o://1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990" gracePeriod=30
Sep 18 16:38:02.738777 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.738430    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kube-rbac-proxy" containerID="cri-o://31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa" gracePeriod=30
Sep 18 16:38:02.815849 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.815819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:02.881634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:02.881517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:03.090070 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:03.090030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:03.217415 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:03.217377    2563 generic.go:356] "Generic (PLEG): container finished" podID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerID="31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa" exitCode=2
Sep 18 16:38:03.217652 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:03.217551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerDied","Data":"31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa"}
Sep 18 16:38:03.224420 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:03.224391    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerID="bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97" exitCode=2
Sep 18 16:38:03.224597 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:03.224461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerDied","Data":"bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97"}
Sep 18 16:38:04.691729 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:04.691683    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 16:38:04.698247 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:04.698219    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 16:38:05.803466 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:05.803427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 16:38:05.808787 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:05.808763    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 16:38:07.073578 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.073544    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:38:07.198261 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198229    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2tnd\" (UniqueName: \"kubernetes.io/projected/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-api-access-t2tnd\") pod \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") "
Sep 18 16:38:07.198261 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-cabundle-cert\") pod \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") "
Sep 18 16:38:07.198614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kserve-provision-location\") pod \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") "
Sep 18 16:38:07.198614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls\") pod \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") "
Sep 18 16:38:07.198614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-rbac-proxy-sar-config\") pod \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\" (UID: \"d569322a-afa8-40df-b5ba-fd4b9b8869bd\") "
Sep 18 16:38:07.198785 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d569322a-afa8-40df-b5ba-fd4b9b8869bd" (UID: "d569322a-afa8-40df-b5ba-fd4b9b8869bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:38:07.198785 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d569322a-afa8-40df-b5ba-fd4b9b8869bd" (UID: "d569322a-afa8-40df-b5ba-fd4b9b8869bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:38:07.198785 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.198763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d569322a-afa8-40df-b5ba-fd4b9b8869bd" (UID: "d569322a-afa8-40df-b5ba-fd4b9b8869bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:38:07.200633 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.200609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-api-access-t2tnd" (OuterVolumeSpecName: "kube-api-access-t2tnd") pod "d569322a-afa8-40df-b5ba-fd4b9b8869bd" (UID: "d569322a-afa8-40df-b5ba-fd4b9b8869bd"). InnerVolumeSpecName "kube-api-access-t2tnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:38:07.200710 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.200692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d569322a-afa8-40df-b5ba-fd4b9b8869bd" (UID: "d569322a-afa8-40df-b5ba-fd4b9b8869bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:38:07.241812 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.241780    2563 generic.go:356] "Generic (PLEG): container finished" podID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerID="1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990" exitCode=0
Sep 18 16:38:07.241939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.241844    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"
Sep 18 16:38:07.241939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.241859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerDied","Data":"1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990"}
Sep 18 16:38:07.241939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.241912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd" event={"ID":"d569322a-afa8-40df-b5ba-fd4b9b8869bd","Type":"ContainerDied","Data":"2fbed06e1ccfbda9a40a69dc0095e1adf61139d50a0788b7f00410ea63e513b4"}
Sep 18 16:38:07.241939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.241940    2563 scope.go:117] "RemoveContainer" containerID="31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa"
Sep 18 16:38:07.250511 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.250486    2563 scope.go:117] "RemoveContainer" containerID="1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990"
Sep 18 16:38:07.257742 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.257724    2563 scope.go:117] "RemoveContainer" containerID="b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3"
Sep 18 16:38:07.263507 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.263487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"]
Sep 18 16:38:07.265490 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.265473    2563 scope.go:117] "RemoveContainer" containerID="31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa"
Sep 18 16:38:07.265909 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:38:07.265840    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa\": container with ID starting with 31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa not found: ID does not exist" containerID="31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa"
Sep 18 16:38:07.265909 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.265879    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa"} err="failed to get container status \"31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa\": rpc error: code = NotFound desc = could not find container \"31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa\": container with ID starting with 31b8bcc89af860252dddc4374d8d884abe6f584244e1aea40ae09465132371aa not found: ID does not exist"
Sep 18 16:38:07.265909 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.265902    2563 scope.go:117] "RemoveContainer" containerID="1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990"
Sep 18 16:38:07.266269 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:38:07.266241    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990\": container with ID starting with 1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990 not found: ID does not exist" containerID="1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990"
Sep 18 16:38:07.266355 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.266277    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990"} err="failed to get container status \"1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990\": rpc error: code = NotFound desc = could not find container \"1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990\": container with ID starting with 1d9e3a5ebdc84376df0e929275b06dca034825c0038fc87f8894c11aae008990 not found: ID does not exist"
Sep 18 16:38:07.266355 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.266299    2563 scope.go:117] "RemoveContainer" containerID="b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3"
Sep 18 16:38:07.266646 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:38:07.266626    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3\": container with ID starting with b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3 not found: ID does not exist" containerID="b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3"
Sep 18 16:38:07.266706 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.266654    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3"} err="failed to get container status \"b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3\": rpc error: code = NotFound desc = could not find container \"b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3\": container with ID starting with b3c06b050e1b8f03d0702675520e79580d598ebb248edaa6a34a84584878a8c3 not found: ID does not exist"
Sep 18 16:38:07.267484 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.267466    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-c8b7d-predictor-57cd759889-hvqrd"]
Sep 18 16:38:07.299491 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.299467    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d569322a-afa8-40df-b5ba-fd4b9b8869bd-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:07.299491 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.299491    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:07.299683 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.299500    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tnd\" (UniqueName: \"kubernetes.io/projected/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kube-api-access-t2tnd\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:07.299683 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.299512    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d569322a-afa8-40df-b5ba-fd4b9b8869bd-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:07.299683 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.299520    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d569322a-afa8-40df-b5ba-fd4b9b8869bd-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:07.737164 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.737140    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:38:07.905728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.905700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls\") pod \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") "
Sep 18 16:38:07.905840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.905744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-cabundle-cert\") pod \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") "
Sep 18 16:38:07.905840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.905790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-rbac-proxy-sar-config\") pod \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") "
Sep 18 16:38:07.905840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.905820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxh6z\" (UniqueName: \"kubernetes.io/projected/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-api-access-bxh6z\") pod \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") "
Sep 18 16:38:07.905840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.905838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kserve-provision-location\") pod \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\" (UID: \"5ebe377b-af8c-4bbf-8641-74af533ca5d5\") "
Sep 18 16:38:07.906225 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.906202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ebe377b-af8c-4bbf-8641-74af533ca5d5" (UID: "5ebe377b-af8c-4bbf-8641-74af533ca5d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:38:07.906225 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.906209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ebe377b-af8c-4bbf-8641-74af533ca5d5" (UID: "5ebe377b-af8c-4bbf-8641-74af533ca5d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:38:07.906367 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.906212    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ebe377b-af8c-4bbf-8641-74af533ca5d5" (UID: "5ebe377b-af8c-4bbf-8641-74af533ca5d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:38:07.907953 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.907934    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ebe377b-af8c-4bbf-8641-74af533ca5d5" (UID: "5ebe377b-af8c-4bbf-8641-74af533ca5d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:38:07.908175 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:07.908149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-api-access-bxh6z" (OuterVolumeSpecName: "kube-api-access-bxh6z") pod "5ebe377b-af8c-4bbf-8641-74af533ca5d5" (UID: "5ebe377b-af8c-4bbf-8641-74af533ca5d5"). InnerVolumeSpecName "kube-api-access-bxh6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:38:08.006775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.006745    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:08.006775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.006773    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:08.006958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.006785    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh6z\" (UniqueName: \"kubernetes.io/projected/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kube-api-access-bxh6z\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:08.006958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.006795    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ebe377b-af8c-4bbf-8641-74af533ca5d5-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:08.006958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.006804    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ebe377b-af8c-4bbf-8641-74af533ca5d5-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:38:08.217423 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.217343    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" path="/var/lib/kubelet/pods/d569322a-afa8-40df-b5ba-fd4b9b8869bd/volumes"
Sep 18 16:38:08.256516 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.256442    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerID="07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e" exitCode=0
Sep 18 16:38:08.256737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.256533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerDied","Data":"07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e"}
Sep 18 16:38:08.256737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.256588    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"
Sep 18 16:38:08.256737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.256705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6" event={"ID":"5ebe377b-af8c-4bbf-8641-74af533ca5d5","Type":"ContainerDied","Data":"58224a1b0fe74f07de04b69fe150998d4f41d3c950874bc7d31d6bc845eb06af"}
Sep 18 16:38:08.256902 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.256750    2563 scope.go:117] "RemoveContainer" containerID="bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97"
Sep 18 16:38:08.265832 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.265815    2563 scope.go:117] "RemoveContainer" containerID="07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e"
Sep 18 16:38:08.273891 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.273873    2563 scope.go:117] "RemoveContainer" containerID="601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c"
Sep 18 16:38:08.274471 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.274450    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"]
Sep 18 16:38:08.278605 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.278583    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-c8b7d-predictor-7c948fbf4b-6s2z6"]
Sep 18 16:38:08.282322 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.282306    2563 scope.go:117] "RemoveContainer" containerID="bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97"
Sep 18 16:38:08.282626 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:38:08.282606    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97\": container with ID starting with bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97 not found: ID does not exist" containerID="bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97"
Sep 18 16:38:08.282686 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.282636    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97"} err="failed to get container status \"bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97\": rpc error: code = NotFound desc = could not find container \"bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97\": container with ID starting with bd09b3a52325b5900a7b7f99eb47fffbc26398412131273070e3172fc642ce97 not found: ID does not exist"
Sep 18 16:38:08.282686 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.282660    2563 scope.go:117] "RemoveContainer" containerID="07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e"
Sep 18 16:38:08.282940 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:38:08.282922    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e\": container with ID starting with 07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e not found: ID does not exist" containerID="07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e"
Sep 18 16:38:08.282998 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.282947    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e"} err="failed to get container status \"07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e\": rpc error: code = NotFound desc = could not find container \"07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e\": container with ID starting with 07ddcc58fc9f14e1328b3b3b8bab54bcaa362610b2161125d959de455d475b6e not found: ID does not exist"
Sep 18 16:38:08.282998 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.282964    2563 scope.go:117] "RemoveContainer" containerID="601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c"
Sep 18 16:38:08.283219 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:38:08.283204    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c\": container with ID starting with 601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c not found: ID does not exist" containerID="601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c"
Sep 18 16:38:08.283276 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:08.283225    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c"} err="failed to get container status \"601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c\": rpc error: code = NotFound desc = could not find container \"601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c\": container with ID starting with 601fdce5975eab2a54050c3296f60237439543ec2ef6d1400ae5ef4e73fe6c9c not found: ID does not exist"
Sep 18 16:38:10.218886 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:10.218854    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" path="/var/lib/kubelet/pods/5ebe377b-af8c-4bbf-8641-74af533ca5d5/volumes"
Sep 18 16:38:22.888643 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.888607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:22.919784 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.919749    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"]
Sep 18 16:38:22.920138 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="storage-initializer"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920141    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="storage-initializer"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920166    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="storage-initializer"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920175    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="storage-initializer"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920184    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920194    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920202    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920210    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920222    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920230    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="storage-initializer"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920251    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="storage-initializer"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920262    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="storage-initializer"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920270    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="storage-initializer"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920280    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920288    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920325    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920340    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920348    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920450    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920464    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920476    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920486    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebe377b-af8c-4bbf-8641-74af533ca5d5" containerName="kserve-container"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920506    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c657f83-3411-489e-a770-b0713ed17508" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920514    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f41ccb-5c02-437e-b3c5-c93205cf3c7d" containerName="kube-rbac-proxy"
Sep 18 16:38:22.920766 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.920523    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d569322a-afa8-40df-b5ba-fd4b9b8869bd" containerName="kserve-container"
Sep 18 16:38:22.923318 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.923299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:22.925508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.925486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-031b1-predictor-serving-cert\""
Sep 18 16:38:22.925640 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.925545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:38:22.925640 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.925618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:38:22.925862 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.925846    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-031b1-kube-rbac-proxy-sar-config\""
Sep 18 16:38:22.926317 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.926300    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:38:22.926366 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.926356    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:38:22.926407 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.926361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:38:22.935271 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.935248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"]
Sep 18 16:38:22.998741 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:22.998711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:38:23.033392 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.033369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmwd\" (UniqueName: \"kubernetes.io/projected/3c8438bb-f470-4473-9b72-c46c989ac175-kube-api-access-jpmwd\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.033521 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.033401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-cabundle-cert\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.033521 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.033420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c8438bb-f470-4473-9b72-c46c989ac175-proxy-tls\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.033521 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.033454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c8438bb-f470-4473-9b72-c46c989ac175-kserve-provision-location\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.033664 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.033518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.133883 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.133850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmwd\" (UniqueName: \"kubernetes.io/projected/3c8438bb-f470-4473-9b72-c46c989ac175-kube-api-access-jpmwd\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.133888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-cabundle-cert\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.133919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c8438bb-f470-4473-9b72-c46c989ac175-proxy-tls\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.133950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c8438bb-f470-4473-9b72-c46c989ac175-kserve-provision-location\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.134007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134412 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.134388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c8438bb-f470-4473-9b72-c46c989ac175-kserve-provision-location\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.134798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.134914 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.134798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-cabundle-cert\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.136732 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.136709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c8438bb-f470-4473-9b72-c46c989ac175-proxy-tls\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.142214 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.142156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmwd\" (UniqueName: \"kubernetes.io/projected/3c8438bb-f470-4473-9b72-c46c989ac175-kube-api-access-jpmwd\") pod \"isvc-logger-raw-031b1-predictor-7f898786d6-ff67l\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") " pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.232776 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.232737    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:23.505056 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:23.504986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"]
Sep 18 16:38:23.507318 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:38:23.507290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8438bb_f470_4473_9b72_c46c989ac175.slice/crio-d3ef05e13e9cb56a1a518d8efb496127273ce9f97d8a979ad6f91af26d3f4120 WatchSource:0}: Error finding container d3ef05e13e9cb56a1a518d8efb496127273ce9f97d8a979ad6f91af26d3f4120: Status 404 returned error can't find the container with id d3ef05e13e9cb56a1a518d8efb496127273ce9f97d8a979ad6f91af26d3f4120
Sep 18 16:38:24.322728 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:24.322687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerStarted","Data":"466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372"}
Sep 18 16:38:24.323109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:24.322773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerStarted","Data":"d3ef05e13e9cb56a1a518d8efb496127273ce9f97d8a979ad6f91af26d3f4120"}
Sep 18 16:38:28.341189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:28.341156    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c8438bb-f470-4473-9b72-c46c989ac175" containerID="466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372" exitCode=0
Sep 18 16:38:28.341189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:28.341195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerDied","Data":"466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372"}
Sep 18 16:38:29.347503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.347457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerStarted","Data":"a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b"}
Sep 18 16:38:29.347916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.347535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerStarted","Data":"7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761"}
Sep 18 16:38:29.347916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.347577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerStarted","Data":"9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129"}
Sep 18 16:38:29.347916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.347791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:29.347916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.347854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:29.349260 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.349232    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:29.368667 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:29.368623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podStartSLOduration=3.461214335 podStartE2EDuration="7.368610736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.907396401s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:38:29.36713561 +0000 UTC m=+1321.367396440" watchObservedRunningTime="2026-09-18 16:38:29.368610736 +0000 UTC m=+1321.368871547"
Sep 18 16:38:30.352052 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:30.352021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:30.352462 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:30.352188    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:30.353266 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:30.353232    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:38:31.356349 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:31.356309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:31.356758 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:31.356699    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:38:36.361029 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:36.360998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:38:36.361698 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:36.361665    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:36.361920 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:36.361896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:38:46.362363 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:46.362260    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:46.362869 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:46.362818    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:38:56.362043 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:56.361995    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:38:56.363654 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:38:56.362504    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:39:06.362415 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:06.362372    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:39:06.362900 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:06.362877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:39:07.387110 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.387080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:07.421522 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.421484    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"]
Sep 18 16:39:07.425350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.425331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.427706 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.427679    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-4e8a5a\""
Sep 18 16:39:07.428496 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.428476    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-4e8a5a-kube-rbac-proxy-sar-config\""
Sep 18 16:39:07.428616 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.428588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-4e8a5a-dockercfg-5wb2f\""
Sep 18 16:39:07.428687 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.428485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-4e8a5a-predictor-serving-cert\""
Sep 18 16:39:07.434917 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.434899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"]
Sep 18 16:39:07.463756 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.463726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:07.500642 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.500611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzw9p\" (UniqueName: \"kubernetes.io/projected/4a559517-4284-4880-b536-7b5729ec1729-kube-api-access-qzw9p\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.500764 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.500665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-cabundle-cert\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.500764 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.500736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.500840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.500773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a559517-4284-4880-b536-7b5729ec1729-kserve-provision-location\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.500874 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.500850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a559517-4284-4880-b536-7b5729ec1729-proxy-tls\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.601857 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.601825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzw9p\" (UniqueName: \"kubernetes.io/projected/4a559517-4284-4880-b536-7b5729ec1729-kube-api-access-qzw9p\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.601866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-cabundle-cert\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.601883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.601898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a559517-4284-4880-b536-7b5729ec1729-kserve-provision-location\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.601959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a559517-4284-4880-b536-7b5729ec1729-proxy-tls\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602378 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.602360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a559517-4284-4880-b536-7b5729ec1729-kserve-provision-location\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602583 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.602544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-cabundle-cert\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.602701 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.602623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.604493 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.604469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a559517-4284-4880-b536-7b5729ec1729-proxy-tls\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.610983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.610966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzw9p\" (UniqueName: \"kubernetes.io/projected/4a559517-4284-4880-b536-7b5729ec1729-kube-api-access-qzw9p\") pod \"isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") " pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:07.736663 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:07.736627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:08.005829 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:08.005806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"]
Sep 18 16:39:08.007962 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:39:08.007934    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a559517_4284_4880_b536_7b5729ec1729.slice/crio-1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6 WatchSource:0}: Error finding container 1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6: Status 404 returned error can't find the container with id 1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6
Sep 18 16:39:08.509648 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:08.509611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" event={"ID":"4a559517-4284-4880-b536-7b5729ec1729","Type":"ContainerStarted","Data":"01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5"}
Sep 18 16:39:08.510063 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:08.509674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" event={"ID":"4a559517-4284-4880-b536-7b5729ec1729","Type":"ContainerStarted","Data":"1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6"}
Sep 18 16:39:13.530269 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:13.530241    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/0.log"
Sep 18 16:39:13.530666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:13.530282    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a559517-4284-4880-b536-7b5729ec1729" containerID="01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5" exitCode=1
Sep 18 16:39:13.530666 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:13.530367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" event={"ID":"4a559517-4284-4880-b536-7b5729ec1729","Type":"ContainerDied","Data":"01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5"}
Sep 18 16:39:14.535927 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:14.535898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/0.log"
Sep 18 16:39:14.536308 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:14.535999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" event={"ID":"4a559517-4284-4880-b536-7b5729ec1729","Type":"ContainerStarted","Data":"584a4571b9e8fa027e65d794a0eeab6cc50e4eaf63348ffd7eb6dc5ade3a57fc"}
Sep 18 16:39:16.362027 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.361923    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:39:16.362443 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.362416    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:39:16.545550 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.545522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/1.log"
Sep 18 16:39:16.545974 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.545959    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/0.log"
Sep 18 16:39:16.546018 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.545992    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a559517-4284-4880-b536-7b5729ec1729" containerID="584a4571b9e8fa027e65d794a0eeab6cc50e4eaf63348ffd7eb6dc5ade3a57fc" exitCode=1
Sep 18 16:39:16.546092 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.546074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" event={"ID":"4a559517-4284-4880-b536-7b5729ec1729","Type":"ContainerDied","Data":"584a4571b9e8fa027e65d794a0eeab6cc50e4eaf63348ffd7eb6dc5ade3a57fc"}
Sep 18 16:39:16.546153 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.546142    2563 scope.go:117] "RemoveContainer" containerID="01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5"
Sep 18 16:39:16.546494 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.546474    2563 scope.go:117] "RemoveContainer" containerID="01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5"
Sep 18 16:39:16.556955 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:39:16.556936    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_kserve-ci-e2e-test(4a559517-4284-4880-b536-7b5729ec1729)\"" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" podUID="4a559517-4284-4880-b536-7b5729ec1729"
Sep 18 16:39:16.557487 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:39:16.557462    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_kserve-ci-e2e-test_4a559517-4284-4880-b536-7b5729ec1729_0 in pod sandbox 1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6 from index: no such id: '01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5'" containerID="01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5"
Sep 18 16:39:16.557550 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:16.557496    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_kserve-ci-e2e-test_4a559517-4284-4880-b536-7b5729ec1729_0 in pod sandbox 1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6 from index: no such id: '01c03f94b4008088d3ffe96464f1d16dc983fc0c7f45ad30c0e492f3374674e5'"
Sep 18 16:39:17.551505 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:17.551477    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/1.log"
Sep 18 16:39:25.456216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.456185    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"]
Sep 18 16:39:25.480876 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.480767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:25.542145 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.542110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:25.585436 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.585409    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/1.log"
Sep 18 16:39:25.587068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.587042    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl_4a559517-4284-4880-b536-7b5729ec1729/storage-initializer/1.log"
Sep 18 16:39:25.587173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.587155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl" event={"ID":"4a559517-4284-4880-b536-7b5729ec1729","Type":"ContainerDied","Data":"1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6"}
Sep 18 16:39:25.587225 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.587204    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ef3e22721097df9c5b312e5f3a4914a631b99a014b84ee36d2a3a36e7b8e2c6"
Sep 18 16:39:25.589025 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.589010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:25.652854 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.652822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzw9p\" (UniqueName: \"kubernetes.io/projected/4a559517-4284-4880-b536-7b5729ec1729-kube-api-access-qzw9p\") pod \"4a559517-4284-4880-b536-7b5729ec1729\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") "
Sep 18 16:39:25.653032 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.652871    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-kube-rbac-proxy-sar-config\") pod \"4a559517-4284-4880-b536-7b5729ec1729\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") "
Sep 18 16:39:25.653032 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.652900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a559517-4284-4880-b536-7b5729ec1729-proxy-tls\") pod \"4a559517-4284-4880-b536-7b5729ec1729\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") "
Sep 18 16:39:25.653032 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.652923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-cabundle-cert\") pod \"4a559517-4284-4880-b536-7b5729ec1729\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") "
Sep 18 16:39:25.653157 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.653038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a559517-4284-4880-b536-7b5729ec1729-kserve-provision-location\") pod \"4a559517-4284-4880-b536-7b5729ec1729\" (UID: \"4a559517-4284-4880-b536-7b5729ec1729\") "
Sep 18 16:39:25.653301 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.653270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a559517-4284-4880-b536-7b5729ec1729" (UID: "4a559517-4284-4880-b536-7b5729ec1729"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:25.653301 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.653287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a559517-4284-4880-b536-7b5729ec1729-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a559517-4284-4880-b536-7b5729ec1729" (UID: "4a559517-4284-4880-b536-7b5729ec1729"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:25.653422 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.653273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a559517-4284-4880-b536-7b5729ec1729" (UID: "4a559517-4284-4880-b536-7b5729ec1729"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:25.655106 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.655083    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a559517-4284-4880-b536-7b5729ec1729-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a559517-4284-4880-b536-7b5729ec1729" (UID: "4a559517-4284-4880-b536-7b5729ec1729"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:25.655215 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.655200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a559517-4284-4880-b536-7b5729ec1729-kube-api-access-qzw9p" (OuterVolumeSpecName: "kube-api-access-qzw9p") pod "4a559517-4284-4880-b536-7b5729ec1729" (UID: "4a559517-4284-4880-b536-7b5729ec1729"). InnerVolumeSpecName "kube-api-access-qzw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:25.753958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.753858    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzw9p\" (UniqueName: \"kubernetes.io/projected/4a559517-4284-4880-b536-7b5729ec1729-kube-api-access-qzw9p\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:25.753958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.753900    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:25.753958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.753911    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a559517-4284-4880-b536-7b5729ec1729-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:25.753958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.753922    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a559517-4284-4880-b536-7b5729ec1729-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:25.753958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:25.753930    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a559517-4284-4880-b536-7b5729ec1729-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:26.361587 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:26.361524    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:39:26.362046 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:26.362022    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:39:26.591919 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:26.591835    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"
Sep 18 16:39:26.623479 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:26.623446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"]
Sep 18 16:39:26.627626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:26.627600    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4e8a5a-predictor-7c5466964-6bnbl"]
Sep 18 16:39:28.218513 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:28.218478    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a559517-4284-4880-b536-7b5729ec1729" path="/var/lib/kubelet/pods/4a559517-4284-4880-b536-7b5729ec1729/volumes"
Sep 18 16:39:30.686348 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.686308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:30.714179 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"]
Sep 18 16:39:30.714485 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714474    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a559517-4284-4880-b536-7b5729ec1729" containerName="storage-initializer"
Sep 18 16:39:30.714534 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714487    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a559517-4284-4880-b536-7b5729ec1729" containerName="storage-initializer"
Sep 18 16:39:30.714594 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714575    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a559517-4284-4880-b536-7b5729ec1729" containerName="storage-initializer"
Sep 18 16:39:30.714594 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714583    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a559517-4284-4880-b536-7b5729ec1729" containerName="storage-initializer"
Sep 18 16:39:30.714662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714648    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a559517-4284-4880-b536-7b5729ec1729" containerName="storage-initializer"
Sep 18 16:39:30.714662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.714654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a559517-4284-4880-b536-7b5729ec1729" containerName="storage-initializer"
Sep 18 16:39:30.718590 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.718552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.720870 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.720845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9d8350-predictor-serving-cert\""
Sep 18 16:39:30.721010 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.720891    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-9d8350-dockercfg-9gt46\""
Sep 18 16:39:30.721010 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.720905    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9d8350-kube-rbac-proxy-sar-config\""
Sep 18 16:39:30.721010 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.720892    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-9d8350\""
Sep 18 16:39:30.728114 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.728087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"]
Sep 18 16:39:30.796679 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.796641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.796846 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.796708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrtw\" (UniqueName: \"kubernetes.io/projected/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-api-access-7lrtw\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.796846 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.796735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kserve-provision-location\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.796846 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.796761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.796846 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.796777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-cabundle-cert\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.897745 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.897716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrtw\" (UniqueName: \"kubernetes.io/projected/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-api-access-7lrtw\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.897946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.897758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kserve-provision-location\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.897946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.897789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.897946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.897813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-cabundle-cert\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.897946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.897847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.898152 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:39:30.897994    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-serving-cert: secret "isvc-init-fail-9d8350-predictor-serving-cert" not found
Sep 18 16:39:30.898152 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:39:30.898056    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls podName:2ff0983a-a461-4ce9-bd7d-8bfee00f99da nodeName:}" failed. No retries permitted until 2026-09-18 16:39:31.398035919 +0000 UTC m=+1383.398296709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls") pod "isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" (UID: "2ff0983a-a461-4ce9-bd7d-8bfee00f99da") : secret "isvc-init-fail-9d8350-predictor-serving-cert" not found
Sep 18 16:39:30.898239 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.898184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kserve-provision-location\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.898459 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.898441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.898495 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.898457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-cabundle-cert\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:30.906404 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:30.906376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrtw\" (UniqueName: \"kubernetes.io/projected/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-api-access-7lrtw\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:31.060029 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:31.059950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:31.402412 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:31.402384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:31.404911 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:31.404890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls\") pod \"isvc-init-fail-9d8350-predictor-8498856cb8-c89fx\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") " pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:31.629480 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:31.629446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:31.908108 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:31.908084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"]
Sep 18 16:39:31.909963 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:39:31.909922    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff0983a_a461_4ce9_bd7d_8bfee00f99da.slice/crio-ca64b00fe5ab39398ff24be52e50c4dc3a45d7818d740baf9990c6a17283646d WatchSource:0}: Error finding container ca64b00fe5ab39398ff24be52e50c4dc3a45d7818d740baf9990c6a17283646d: Status 404 returned error can't find the container with id ca64b00fe5ab39398ff24be52e50c4dc3a45d7818d740baf9990c6a17283646d
Sep 18 16:39:32.619634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:32.619595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" event={"ID":"2ff0983a-a461-4ce9-bd7d-8bfee00f99da","Type":"ContainerStarted","Data":"184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451"}
Sep 18 16:39:32.619817 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:32.619661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" event={"ID":"2ff0983a-a461-4ce9-bd7d-8bfee00f99da","Type":"ContainerStarted","Data":"ca64b00fe5ab39398ff24be52e50c4dc3a45d7818d740baf9990c6a17283646d"}
Sep 18 16:39:36.362053 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:36.361950    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:39:36.362517 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:36.362450    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:39:36.637374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:36.637290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9d8350-predictor-8498856cb8-c89fx_2ff0983a-a461-4ce9-bd7d-8bfee00f99da/storage-initializer/0.log"
Sep 18 16:39:36.637374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:36.637328    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerID="184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451" exitCode=1
Sep 18 16:39:36.637618 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:36.637409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" event={"ID":"2ff0983a-a461-4ce9-bd7d-8bfee00f99da","Type":"ContainerDied","Data":"184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451"}
Sep 18 16:39:37.643757 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:37.643729    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9d8350-predictor-8498856cb8-c89fx_2ff0983a-a461-4ce9-bd7d-8bfee00f99da/storage-initializer/0.log"
Sep 18 16:39:37.644143 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:37.643776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" event={"ID":"2ff0983a-a461-4ce9-bd7d-8bfee00f99da","Type":"ContainerStarted","Data":"feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec"}
Sep 18 16:39:40.753937 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:40.753886    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"]
Sep 18 16:39:40.754448 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:40.754094    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer" containerID="cri-o://feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec" gracePeriod=30
Sep 18 16:39:40.788227 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:40.788190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:41.496295 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.496273    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9d8350-predictor-8498856cb8-c89fx_2ff0983a-a461-4ce9-bd7d-8bfee00f99da/storage-initializer/1.log"
Sep 18 16:39:41.496682 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.496667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9d8350-predictor-8498856cb8-c89fx_2ff0983a-a461-4ce9-bd7d-8bfee00f99da/storage-initializer/0.log"
Sep 18 16:39:41.498825 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.498811    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:41.593746 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.593721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kserve-provision-location\") pod \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") "
Sep 18 16:39:41.593912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.593771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls\") pod \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") "
Sep 18 16:39:41.593912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.593787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-cabundle-cert\") pod \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") "
Sep 18 16:39:41.593912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.593847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-rbac-proxy-sar-config\") pod \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") "
Sep 18 16:39:41.593912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.593886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lrtw\" (UniqueName: \"kubernetes.io/projected/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-api-access-7lrtw\") pod \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\" (UID: \"2ff0983a-a461-4ce9-bd7d-8bfee00f99da\") "
Sep 18 16:39:41.594119 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.594024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ff0983a-a461-4ce9-bd7d-8bfee00f99da" (UID: "2ff0983a-a461-4ce9-bd7d-8bfee00f99da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:41.594195 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.594174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ff0983a-a461-4ce9-bd7d-8bfee00f99da" (UID: "2ff0983a-a461-4ce9-bd7d-8bfee00f99da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:41.594232 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.594173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ff0983a-a461-4ce9-bd7d-8bfee00f99da" (UID: "2ff0983a-a461-4ce9-bd7d-8bfee00f99da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:39:41.596002 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.595978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-api-access-7lrtw" (OuterVolumeSpecName: "kube-api-access-7lrtw") pod "2ff0983a-a461-4ce9-bd7d-8bfee00f99da" (UID: "2ff0983a-a461-4ce9-bd7d-8bfee00f99da"). InnerVolumeSpecName "kube-api-access-7lrtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:41.596103 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.596016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ff0983a-a461-4ce9-bd7d-8bfee00f99da" (UID: "2ff0983a-a461-4ce9-bd7d-8bfee00f99da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:41.660980 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.660952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9d8350-predictor-8498856cb8-c89fx_2ff0983a-a461-4ce9-bd7d-8bfee00f99da/storage-initializer/1.log"
Sep 18 16:39:41.661335 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.661321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9d8350-predictor-8498856cb8-c89fx_2ff0983a-a461-4ce9-bd7d-8bfee00f99da/storage-initializer/0.log"
Sep 18 16:39:41.661379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.661358    2563 generic.go:356] "Generic (PLEG): container finished" podID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerID="feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec" exitCode=1
Sep 18 16:39:41.661414 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.661389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" event={"ID":"2ff0983a-a461-4ce9-bd7d-8bfee00f99da","Type":"ContainerDied","Data":"feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec"}
Sep 18 16:39:41.661453 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.661420    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"
Sep 18 16:39:41.661487 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.661453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx" event={"ID":"2ff0983a-a461-4ce9-bd7d-8bfee00f99da","Type":"ContainerDied","Data":"ca64b00fe5ab39398ff24be52e50c4dc3a45d7818d740baf9990c6a17283646d"}
Sep 18 16:39:41.661518 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.661490    2563 scope.go:117] "RemoveContainer" containerID="feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec"
Sep 18 16:39:41.670356 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.670320    2563 scope.go:117] "RemoveContainer" containerID="184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451"
Sep 18 16:39:41.678028 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.678012    2563 scope.go:117] "RemoveContainer" containerID="feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec"
Sep 18 16:39:41.678314 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:39:41.678295    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec\": container with ID starting with feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec not found: ID does not exist" containerID="feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec"
Sep 18 16:39:41.678362 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.678324    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec"} err="failed to get container status \"feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec\": rpc error: code = NotFound desc = could not find container \"feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec\": container with ID starting with feb4f9f28a735d71687b9f31d04511d8782fbc0783f4da7b12a999d837565cec not found: ID does not exist"
Sep 18 16:39:41.678362 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.678340    2563 scope.go:117] "RemoveContainer" containerID="184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451"
Sep 18 16:39:41.678616 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:39:41.678593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451\": container with ID starting with 184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451 not found: ID does not exist" containerID="184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451"
Sep 18 16:39:41.678676 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.678625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451"} err="failed to get container status \"184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451\": rpc error: code = NotFound desc = could not find container \"184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451\": container with ID starting with 184e6f2c6cd7df87270ea4d64055585d875e62f9e73c97544f4e22c520add451 not found: ID does not exist"
Sep 18 16:39:41.694681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.694659    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:41.694681 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.694683    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrtw\" (UniqueName: \"kubernetes.io/projected/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kube-api-access-7lrtw\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:41.694811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.694692    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:41.694811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.694701    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:41.694811 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.694712    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ff0983a-a461-4ce9-bd7d-8bfee00f99da-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:39:41.700396 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.700376    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"]
Sep 18 16:39:41.705031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:41.705012    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9d8350-predictor-8498856cb8-c89fx"]
Sep 18 16:39:42.217981 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:42.217949    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" path="/var/lib/kubelet/pods/2ff0983a-a461-4ce9-bd7d-8bfee00f99da/volumes"
Sep 18 16:39:45.959147 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:45.959117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:46.001597 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001542    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"]
Sep 18 16:39:46.001907 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer"
Sep 18 16:39:46.001963 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001908    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer"
Sep 18 16:39:46.001963 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001917    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer"
Sep 18 16:39:46.001963 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001922    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer"
Sep 18 16:39:46.002062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001980    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer"
Sep 18 16:39:46.002062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.001991    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff0983a-a461-4ce9-bd7d-8bfee00f99da" containerName="storage-initializer"
Sep 18 16:39:46.005231 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.005214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.008452 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.008426    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-723b8-kube-rbac-proxy-sar-config\""
Sep 18 16:39:46.008601 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.008506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-723b8-predictor-serving-cert\""
Sep 18 16:39:46.019377 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.019353    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"]
Sep 18 16:39:46.048037 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.048000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:46.126342 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.126311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-proxy-tls\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.126342 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.126348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-cabundle-cert\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.126619 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.126381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7bw\" (UniqueName: \"kubernetes.io/projected/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-api-access-rp7bw\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.126619 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.126474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.126619 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.126608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kserve-provision-location\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.227650 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.227551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7bw\" (UniqueName: \"kubernetes.io/projected/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-api-access-rp7bw\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.227650 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.227619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.227856 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.227692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kserve-provision-location\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.227856 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.227750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-proxy-tls\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.227856 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.227773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-cabundle-cert\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.228234 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.228214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kserve-provision-location\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.228300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.228276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.228350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.228336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-cabundle-cert\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.230382 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.230355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-proxy-tls\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.236208 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.236187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7bw\" (UniqueName: \"kubernetes.io/projected/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-api-access-rp7bw\") pod \"raw-sklearn-723b8-predictor-66b5f9879d-nssdq\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") " pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.316322 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.316281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:46.362735 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.362703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:39:46.362969 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.362953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:39:46.591871 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.591846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"]
Sep 18 16:39:46.594228 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:39:46.594196    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eba24bc_a9a7_416e_93f6_d623ae0ae64b.slice/crio-5929d1f8e8409deb10e353cb01070864a1fcc6e5f1a21257b46cecd69b2f1077 WatchSource:0}: Error finding container 5929d1f8e8409deb10e353cb01070864a1fcc6e5f1a21257b46cecd69b2f1077: Status 404 returned error can't find the container with id 5929d1f8e8409deb10e353cb01070864a1fcc6e5f1a21257b46cecd69b2f1077
Sep 18 16:39:46.690289 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.690255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerStarted","Data":"c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0"}
Sep 18 16:39:46.690418 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:46.690312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerStarted","Data":"5929d1f8e8409deb10e353cb01070864a1fcc6e5f1a21257b46cecd69b2f1077"}
Sep 18 16:39:49.374914 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:49.374880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:50.709694 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:50.709648    2563 generic.go:356] "Generic (PLEG): container finished" podID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerID="c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0" exitCode=0
Sep 18 16:39:50.710088 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:50.709720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerDied","Data":"c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0"}
Sep 18 16:39:51.715763 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:51.715723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerStarted","Data":"d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be"}
Sep 18 16:39:51.716240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:51.715796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerStarted","Data":"f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b"}
Sep 18 16:39:51.716240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:51.716014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:52.720535 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:52.720507    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:52.721950 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:52.721919    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:39:52.737944 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:52.737902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podStartSLOduration=4.104902944 podStartE2EDuration="7.737887209s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632984265s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:52.736245285 +0000 UTC m=+1404.736506095" watchObservedRunningTime="2026-09-18 16:39:52.737887209 +0000 UTC m=+1404.738148020"
Sep 18 16:39:53.725139 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:53.725096    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:39:58.625387 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.625353    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"]
Sep 18 16:39:58.625841 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.625726    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" containerID="cri-o://9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129" gracePeriod=30
Sep 18 16:39:58.625841 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.625755    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" containerID="cri-o://a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b" gracePeriod=30
Sep 18 16:39:58.625975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.625769    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" containerID="cri-o://7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761" gracePeriod=30
Sep 18 16:39:58.729921 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.729889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:39:58.730409 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.730364    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:39:58.747994 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.747971    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c8438bb-f470-4473-9b72-c46c989ac175" containerID="7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761" exitCode=2
Sep 18 16:39:58.748090 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:58.748030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerDied","Data":"7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761"}
Sep 18 16:39:59.488543 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:59.488504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:59.587352 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:59.587308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:39:59.688071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:39:59.688033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:01.356810 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:01.356751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:40:03.776715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:03.776684    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c8438bb-f470-4473-9b72-c46c989ac175" containerID="9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129" exitCode=0
Sep 18 16:40:03.777078 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:03.776761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerDied","Data":"9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129"}
Sep 18 16:40:06.357365 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:06.357323    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:40:06.361929 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:06.361886    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:40:06.362215 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:06.362190    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:40:08.731295 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:08.731203    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:40:11.357008 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:11.356963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:40:11.357386 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:11.357094    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:40:16.356821 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:16.356776    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:40:16.362192 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:16.362156    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:40:16.362476 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:16.362431    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:40:18.731173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:18.731135    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:40:21.356739 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:21.356694    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:40:25.651228 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:25.651197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:40:26.356862 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:26.356819    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 16:40:26.362291 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:26.362243    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 16:40:26.362447 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:26.362362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:40:26.362625 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:26.362588    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:40:26.362705 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:26.362691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:40:28.730798 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.730758    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:40:28.772065 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.772041    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:40:28.802741 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.802711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c8438bb-f470-4473-9b72-c46c989ac175-proxy-tls\") pod \"3c8438bb-f470-4473-9b72-c46c989ac175\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") "
Sep 18 16:40:28.802741 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.802746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-kube-rbac-proxy-sar-config\") pod \"3c8438bb-f470-4473-9b72-c46c989ac175\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") "
Sep 18 16:40:28.802988 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.802788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c8438bb-f470-4473-9b72-c46c989ac175-kserve-provision-location\") pod \"3c8438bb-f470-4473-9b72-c46c989ac175\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") "
Sep 18 16:40:28.802988 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.802945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-cabundle-cert\") pod \"3c8438bb-f470-4473-9b72-c46c989ac175\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") "
Sep 18 16:40:28.803109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.802995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpmwd\" (UniqueName: \"kubernetes.io/projected/3c8438bb-f470-4473-9b72-c46c989ac175-kube-api-access-jpmwd\") pod \"3c8438bb-f470-4473-9b72-c46c989ac175\" (UID: \"3c8438bb-f470-4473-9b72-c46c989ac175\") "
Sep 18 16:40:28.803172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.803117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8438bb-f470-4473-9b72-c46c989ac175-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c8438bb-f470-4473-9b72-c46c989ac175" (UID: "3c8438bb-f470-4473-9b72-c46c989ac175"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:28.803172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.803118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c8438bb-f470-4473-9b72-c46c989ac175" (UID: "3c8438bb-f470-4473-9b72-c46c989ac175"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:40:28.803307 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.803290    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:40:28.803368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.803308    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c8438bb-f470-4473-9b72-c46c989ac175-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:40:28.803368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.803308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c8438bb-f470-4473-9b72-c46c989ac175" (UID: "3c8438bb-f470-4473-9b72-c46c989ac175"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:40:28.805033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.805008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8438bb-f470-4473-9b72-c46c989ac175-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c8438bb-f470-4473-9b72-c46c989ac175" (UID: "3c8438bb-f470-4473-9b72-c46c989ac175"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:40:28.805207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.805188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8438bb-f470-4473-9b72-c46c989ac175-kube-api-access-jpmwd" (OuterVolumeSpecName: "kube-api-access-jpmwd") pod "3c8438bb-f470-4473-9b72-c46c989ac175" (UID: "3c8438bb-f470-4473-9b72-c46c989ac175"). InnerVolumeSpecName "kube-api-access-jpmwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:40:28.889515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.889481    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c8438bb-f470-4473-9b72-c46c989ac175" containerID="a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b" exitCode=0
Sep 18 16:40:28.889692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.889542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerDied","Data":"a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b"}
Sep 18 16:40:28.889692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.889547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"
Sep 18 16:40:28.889692 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.889651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l" event={"ID":"3c8438bb-f470-4473-9b72-c46c989ac175","Type":"ContainerDied","Data":"d3ef05e13e9cb56a1a518d8efb496127273ce9f97d8a979ad6f91af26d3f4120"}
Sep 18 16:40:28.889799 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.889696    2563 scope.go:117] "RemoveContainer" containerID="a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b"
Sep 18 16:40:28.898253 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.898236    2563 scope.go:117] "RemoveContainer" containerID="7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761"
Sep 18 16:40:28.904429 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.904390    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c8438bb-f470-4473-9b72-c46c989ac175-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:40:28.904533 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.904435    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpmwd\" (UniqueName: \"kubernetes.io/projected/3c8438bb-f470-4473-9b72-c46c989ac175-kube-api-access-jpmwd\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:40:28.904533 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.904457    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c8438bb-f470-4473-9b72-c46c989ac175-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:40:28.910117 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.910095    2563 scope.go:117] "RemoveContainer" containerID="9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129"
Sep 18 16:40:28.911187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.911140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"]
Sep 18 16:40:28.913760 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.913712    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-031b1-predictor-7f898786d6-ff67l"]
Sep 18 16:40:28.918307 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.918291    2563 scope.go:117] "RemoveContainer" containerID="466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372"
Sep 18 16:40:28.927398 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.927366    2563 scope.go:117] "RemoveContainer" containerID="a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b"
Sep 18 16:40:28.927889 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:40:28.927870    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b\": container with ID starting with a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b not found: ID does not exist" containerID="a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b"
Sep 18 16:40:28.927963 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.927899    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b"} err="failed to get container status \"a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b\": rpc error: code = NotFound desc = could not find container \"a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b\": container with ID starting with a01b4137771b69f108d73d37f97661b4e4fbe0a4b57755f6e1c4c335fccbe33b not found: ID does not exist"
Sep 18 16:40:28.927963 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.927917    2563 scope.go:117] "RemoveContainer" containerID="7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761"
Sep 18 16:40:28.928187 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:40:28.928169    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761\": container with ID starting with 7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761 not found: ID does not exist" containerID="7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761"
Sep 18 16:40:28.928259 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.928196    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761"} err="failed to get container status \"7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761\": rpc error: code = NotFound desc = could not find container \"7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761\": container with ID starting with 7aa428d3cc5525568689dc21612040894361fccc0e24a939371416583e842761 not found: ID does not exist"
Sep 18 16:40:28.928259 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.928216    2563 scope.go:117] "RemoveContainer" containerID="9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129"
Sep 18 16:40:28.928509 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:40:28.928491    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129\": container with ID starting with 9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129 not found: ID does not exist" containerID="9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129"
Sep 18 16:40:28.928554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.928514    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129"} err="failed to get container status \"9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129\": rpc error: code = NotFound desc = could not find container \"9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129\": container with ID starting with 9658e901adaf6113f61f69bf79b8bbfef9ef1553ea237581a1be6d15eacbb129 not found: ID does not exist"
Sep 18 16:40:28.928554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.928534    2563 scope.go:117] "RemoveContainer" containerID="466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372"
Sep 18 16:40:28.928795 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:40:28.928779    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372\": container with ID starting with 466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372 not found: ID does not exist" containerID="466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372"
Sep 18 16:40:28.928840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:28.928799    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372"} err="failed to get container status \"466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372\": rpc error: code = NotFound desc = could not find container \"466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372\": container with ID starting with 466c74a4dc9f5099e6d10917ad9377db468396b4f8f2da242d5769bc0f9d4372 not found: ID does not exist"
Sep 18 16:40:30.218104 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:30.218072    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" path="/var/lib/kubelet/pods/3c8438bb-f470-4473-9b72-c46c989ac175/volumes"
Sep 18 16:40:38.730867 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:38.730823    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:40:48.730522 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:48.730483    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:40:58.730656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:40:58.730617    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:41:08.731787 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:08.731755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:41:16.504147 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:16.504114    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"]
Sep 18 16:41:16.504673 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:16.504381    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" containerID="cri-o://f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b" gracePeriod=30
Sep 18 16:41:16.504673 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:16.504403    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kube-rbac-proxy" containerID="cri-o://d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be" gracePeriod=30
Sep 18 16:41:16.527868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:16.527835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:17.087191 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:17.087158    2563 generic.go:356] "Generic (PLEG): container finished" podID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerID="d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be" exitCode=2
Sep 18 16:41:17.087375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:17.087236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerDied","Data":"d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be"}
Sep 18 16:41:18.725943 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:18.725898    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 18 16:41:18.731176 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:18.731150    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 16:41:21.451575 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.451544    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:41:21.533868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.533835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kserve-provision-location\") pod \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") "
Sep 18 16:41:21.533868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.533874    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-proxy-tls\") pod \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") "
Sep 18 16:41:21.534087 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.533906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-rbac-proxy-sar-config\") pod \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") "
Sep 18 16:41:21.534087 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.533928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp7bw\" (UniqueName: \"kubernetes.io/projected/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-api-access-rp7bw\") pod \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") "
Sep 18 16:41:21.534087 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.533984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-cabundle-cert\") pod \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\" (UID: \"4eba24bc-a9a7-416e-93f6-d623ae0ae64b\") "
Sep 18 16:41:21.534252 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.534133    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4eba24bc-a9a7-416e-93f6-d623ae0ae64b" (UID: "4eba24bc-a9a7-416e-93f6-d623ae0ae64b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:41:21.534311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.534289    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:41:21.534368 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.534349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4eba24bc-a9a7-416e-93f6-d623ae0ae64b" (UID: "4eba24bc-a9a7-416e-93f6-d623ae0ae64b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:41:21.534425 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.534375    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4eba24bc-a9a7-416e-93f6-d623ae0ae64b" (UID: "4eba24bc-a9a7-416e-93f6-d623ae0ae64b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:41:21.536143 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.536108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-api-access-rp7bw" (OuterVolumeSpecName: "kube-api-access-rp7bw") pod "4eba24bc-a9a7-416e-93f6-d623ae0ae64b" (UID: "4eba24bc-a9a7-416e-93f6-d623ae0ae64b"). InnerVolumeSpecName "kube-api-access-rp7bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:41:21.536272 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.536255    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4eba24bc-a9a7-416e-93f6-d623ae0ae64b" (UID: "4eba24bc-a9a7-416e-93f6-d623ae0ae64b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:41:21.635250 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.635218    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:41:21.635250 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.635248    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:41:21.635441 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.635268    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7bw\" (UniqueName: \"kubernetes.io/projected/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-kube-api-access-rp7bw\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:41:21.635441 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.635278    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eba24bc-a9a7-416e-93f6-d623ae0ae64b-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:41:21.676118 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.676085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:21.708208 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708176    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"]
Sep 18 16:41:21.708476 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708463    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container"
Sep 18 16:41:21.708476 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708487    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708493    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708498    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="storage-initializer"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708504    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="storage-initializer"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708523    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708533    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="storage-initializer"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708551    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="storage-initializer"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708569    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kube-rbac-proxy"
Sep 18 16:41:21.708585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708575    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kube-rbac-proxy"
Sep 18 16:41:21.708983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708623    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kube-rbac-proxy"
Sep 18 16:41:21.708983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708630    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="kserve-container"
Sep 18 16:41:21.708983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8438bb-f470-4473-9b72-c46c989ac175" containerName="agent"
Sep 18 16:41:21.708983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708644    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kserve-container"
Sep 18 16:41:21.708983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.708649    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerName="kube-rbac-proxy"
Sep 18 16:41:21.711938 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.711922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.714688 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.714669    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-58298-predictor-serving-cert\""
Sep 18 16:41:21.714885 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.714873    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-58298-kube-rbac-proxy-sar-config\""
Sep 18 16:41:21.723483 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.723458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"]
Sep 18 16:41:21.735765 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.735733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7x6c\" (UniqueName: \"kubernetes.io/projected/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-api-access-h7x6c\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.735879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.735817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/910f5941-54c7-4a6e-bb72-24c9b100edab-kserve-provision-location\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.735879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.735867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.735974 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.735912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/910f5941-54c7-4a6e-bb72-24c9b100edab-proxy-tls\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.735974 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.735948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-cabundle-cert\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.836607 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.836518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/910f5941-54c7-4a6e-bb72-24c9b100edab-proxy-tls\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.836607 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.836573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-cabundle-cert\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.836607 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.836603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7x6c\" (UniqueName: \"kubernetes.io/projected/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-api-access-h7x6c\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.836875 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.836636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/910f5941-54c7-4a6e-bb72-24c9b100edab-kserve-provision-location\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.836875 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.836663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.837204 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.837177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/910f5941-54c7-4a6e-bb72-24c9b100edab-kserve-provision-location\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.837297 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.837286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-cabundle-cert\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.837375 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.837358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.839187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.839165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/910f5941-54c7-4a6e-bb72-24c9b100edab-proxy-tls\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.844226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.844203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7x6c\" (UniqueName: \"kubernetes.io/projected/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-api-access-h7x6c\") pod \"raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:21.874602 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:21.874576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:41:22.022573 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.022529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:22.108789 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.108718    2563 generic.go:356] "Generic (PLEG): container finished" podID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" containerID="f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b" exitCode=0
Sep 18 16:41:22.108916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.108802    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"
Sep 18 16:41:22.108916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.108799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerDied","Data":"f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b"}
Sep 18 16:41:22.108916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.108868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq" event={"ID":"4eba24bc-a9a7-416e-93f6-d623ae0ae64b","Type":"ContainerDied","Data":"5929d1f8e8409deb10e353cb01070864a1fcc6e5f1a21257b46cecd69b2f1077"}
Sep 18 16:41:22.108916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.108909    2563 scope.go:117] "RemoveContainer" containerID="d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be"
Sep 18 16:41:22.117583 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.117548    2563 scope.go:117] "RemoveContainer" containerID="f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b"
Sep 18 16:41:22.126737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.126719    2563 scope.go:117] "RemoveContainer" containerID="c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0"
Sep 18 16:41:22.133020 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.132997    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"]
Sep 18 16:41:22.134333 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.134318    2563 scope.go:117] "RemoveContainer" containerID="d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be"
Sep 18 16:41:22.134616 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:41:22.134598    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be\": container with ID starting with d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be not found: ID does not exist" containerID="d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be"
Sep 18 16:41:22.134687 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.134621    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be"} err="failed to get container status \"d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be\": rpc error: code = NotFound desc = could not find container \"d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be\": container with ID starting with d08fe45d56850098d667a98b5d5a2b9f230811fd1e0c37ab6851c00a775191be not found: ID does not exist"
Sep 18 16:41:22.134687 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.134637    2563 scope.go:117] "RemoveContainer" containerID="f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b"
Sep 18 16:41:22.134982 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:41:22.134961    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b\": container with ID starting with f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b not found: ID does not exist" containerID="f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b"
Sep 18 16:41:22.135054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.134987    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b"} err="failed to get container status \"f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b\": rpc error: code = NotFound desc = could not find container \"f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b\": container with ID starting with f3d8952e303cad004742008742e9f1a5bba616a7d94f81188c22d35571170f8b not found: ID does not exist"
Sep 18 16:41:22.135054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.135006    2563 scope.go:117] "RemoveContainer" containerID="c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0"
Sep 18 16:41:22.135372 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:41:22.135346    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0\": container with ID starting with c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0 not found: ID does not exist" containerID="c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0"
Sep 18 16:41:22.135467 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.135375    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0"} err="failed to get container status \"c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0\": rpc error: code = NotFound desc = could not find container \"c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0\": container with ID starting with c32694cbb444c1e1e4fa3928e33212a67555e44ec310cded9d23293491d003d0 not found: ID does not exist"
Sep 18 16:41:22.137161 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.137142    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-723b8-predictor-66b5f9879d-nssdq"]
Sep 18 16:41:22.217948 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.217917    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eba24bc-a9a7-416e-93f6-d623ae0ae64b" path="/var/lib/kubelet/pods/4eba24bc-a9a7-416e-93f6-d623ae0ae64b/volumes"
Sep 18 16:41:22.296879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:22.296843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"]
Sep 18 16:41:22.300242 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:41:22.300217    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910f5941_54c7_4a6e_bb72_24c9b100edab.slice/crio-7f52ff86c5d1b9a9cac0821a3b89f9fb641d30a7da47dae74a37181adcb9b548 WatchSource:0}: Error finding container 7f52ff86c5d1b9a9cac0821a3b89f9fb641d30a7da47dae74a37181adcb9b548: Status 404 returned error can't find the container with id 7f52ff86c5d1b9a9cac0821a3b89f9fb641d30a7da47dae74a37181adcb9b548
Sep 18 16:41:23.115304 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:23.115271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerStarted","Data":"b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093"}
Sep 18 16:41:23.115714 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:23.115326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerStarted","Data":"7f52ff86c5d1b9a9cac0821a3b89f9fb641d30a7da47dae74a37181adcb9b548"}
Sep 18 16:41:26.129298 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:26.129265    2563 generic.go:356] "Generic (PLEG): container finished" podID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerID="b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093" exitCode=0
Sep 18 16:41:26.129682 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:26.129340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerDied","Data":"b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093"}
Sep 18 16:41:27.135230 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:27.135196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerStarted","Data":"46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25"}
Sep 18 16:41:27.135659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:27.135284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerStarted","Data":"606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e"}
Sep 18 16:41:27.135659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:27.135528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:27.160397 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:27.160346    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podStartSLOduration=2.6005209860000003 podStartE2EDuration="6.160330327s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.559809341s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:41:27.157997492 +0000 UTC m=+1499.158258312" watchObservedRunningTime="2026-09-18 16:41:27.160330327 +0000 UTC m=+1499.160591139"
Sep 18 16:41:28.139800 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:28.139755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:28.141022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:28.140999    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:41:29.144178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:29.144140    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:41:30.184158 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:30.184133    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:41:30.184626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:30.184430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:41:34.149237 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:34.149197    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:41:34.154238 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:34.154169    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:41:44.151664 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:44.151551    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:41:54.150836 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:41:54.150797    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:04.150834 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:04.150791    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:14.151020 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:14.150975    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:15.507576 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:15.507526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:24.151610 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:24.151548    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:34.151180 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:34.151139    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:36.212888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:36.212847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:45.189100 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:45.189066    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:46.217394 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:46.217369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:42:52.264226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:52.264189    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"]
Sep 18 16:42:52.264703 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:52.264500    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" containerID="cri-o://606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e" gracePeriod=30
Sep 18 16:42:52.264703 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:52.264604    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kube-rbac-proxy" containerID="cri-o://46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25" gracePeriod=30
Sep 18 16:42:52.292184 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:52.292155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:52.490077 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:52.490034    2563 generic.go:356] "Generic (PLEG): container finished" podID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerID="46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25" exitCode=2
Sep 18 16:42:52.490243 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:52.490090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerDied","Data":"46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25"}
Sep 18 16:42:54.145288 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:54.145247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 18 16:42:56.213189 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:56.213136    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 18 16:42:57.203520 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.203497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:42:57.329405 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/910f5941-54c7-4a6e-bb72-24c9b100edab-proxy-tls\") pod \"910f5941-54c7-4a6e-bb72-24c9b100edab\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") "
Sep 18 16:42:57.329405 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329367    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7x6c\" (UniqueName: \"kubernetes.io/projected/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-api-access-h7x6c\") pod \"910f5941-54c7-4a6e-bb72-24c9b100edab\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") "
Sep 18 16:42:57.329941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-rbac-proxy-sar-config\") pod \"910f5941-54c7-4a6e-bb72-24c9b100edab\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") "
Sep 18 16:42:57.329941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-cabundle-cert\") pod \"910f5941-54c7-4a6e-bb72-24c9b100edab\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") "
Sep 18 16:42:57.329941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/910f5941-54c7-4a6e-bb72-24c9b100edab-kserve-provision-location\") pod \"910f5941-54c7-4a6e-bb72-24c9b100edab\" (UID: \"910f5941-54c7-4a6e-bb72-24c9b100edab\") "
Sep 18 16:42:57.329941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329868    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "910f5941-54c7-4a6e-bb72-24c9b100edab" (UID: "910f5941-54c7-4a6e-bb72-24c9b100edab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:42:57.329941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "910f5941-54c7-4a6e-bb72-24c9b100edab" (UID: "910f5941-54c7-4a6e-bb72-24c9b100edab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:42:57.329941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.329929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910f5941-54c7-4a6e-bb72-24c9b100edab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "910f5941-54c7-4a6e-bb72-24c9b100edab" (UID: "910f5941-54c7-4a6e-bb72-24c9b100edab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:57.331662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.331625    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-api-access-h7x6c" (OuterVolumeSpecName: "kube-api-access-h7x6c") pod "910f5941-54c7-4a6e-bb72-24c9b100edab" (UID: "910f5941-54c7-4a6e-bb72-24c9b100edab"). InnerVolumeSpecName "kube-api-access-h7x6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:42:57.331662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.331633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910f5941-54c7-4a6e-bb72-24c9b100edab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "910f5941-54c7-4a6e-bb72-24c9b100edab" (UID: "910f5941-54c7-4a6e-bb72-24c9b100edab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:42:57.422580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.422532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:57.430656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.430627    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:42:57.430803 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.430658    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/910f5941-54c7-4a6e-bb72-24c9b100edab-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:42:57.430803 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.430690    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/910f5941-54c7-4a6e-bb72-24c9b100edab-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:42:57.430803 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.430703    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/910f5941-54c7-4a6e-bb72-24c9b100edab-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:42:57.430803 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.430719    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7x6c\" (UniqueName: \"kubernetes.io/projected/910f5941-54c7-4a6e-bb72-24c9b100edab-kube-api-access-h7x6c\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:42:57.488413 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488383    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"]
Sep 18 16:42:57.488715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container"
Sep 18 16:42:57.488783 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container"
Sep 18 16:42:57.488783 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488727    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="storage-initializer"
Sep 18 16:42:57.488783 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="storage-initializer"
Sep 18 16:42:57.488783 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488747    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kube-rbac-proxy"
Sep 18 16:42:57.488783 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488752    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kube-rbac-proxy"
Sep 18 16:42:57.488961 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kube-rbac-proxy"
Sep 18 16:42:57.488961 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.488822    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerName="kserve-container"
Sep 18 16:42:57.492291 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.492272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.494805 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.494785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 16:42:57.494908 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.494802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 16:42:57.498289 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.498266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:57.502129 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.502105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"]
Sep 18 16:42:57.513235 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.513208    2563 generic.go:356] "Generic (PLEG): container finished" podID="910f5941-54c7-4a6e-bb72-24c9b100edab" containerID="606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e" exitCode=0
Sep 18 16:42:57.513334 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.513279    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"
Sep 18 16:42:57.513334 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.513288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerDied","Data":"606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e"}
Sep 18 16:42:57.513406 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.513343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527" event={"ID":"910f5941-54c7-4a6e-bb72-24c9b100edab","Type":"ContainerDied","Data":"7f52ff86c5d1b9a9cac0821a3b89f9fb641d30a7da47dae74a37181adcb9b548"}
Sep 18 16:42:57.513406 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.513372    2563 scope.go:117] "RemoveContainer" containerID="46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25"
Sep 18 16:42:57.523143 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.523125    2563 scope.go:117] "RemoveContainer" containerID="606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e"
Sep 18 16:42:57.530446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.530430    2563 scope.go:117] "RemoveContainer" containerID="b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093"
Sep 18 16:42:57.536959 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.536938    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"]
Sep 18 16:42:57.538392 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.538375    2563 scope.go:117] "RemoveContainer" containerID="46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25"
Sep 18 16:42:57.538692 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:42:57.538674    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25\": container with ID starting with 46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25 not found: ID does not exist" containerID="46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25"
Sep 18 16:42:57.538762 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.538704    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25"} err="failed to get container status \"46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25\": rpc error: code = NotFound desc = could not find container \"46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25\": container with ID starting with 46c1ae106b374674c0db025b976e07b94aea8c153d5286fa15d62d7040a39d25 not found: ID does not exist"
Sep 18 16:42:57.538762 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.538727    2563 scope.go:117] "RemoveContainer" containerID="606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e"
Sep 18 16:42:57.538981 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:42:57.538963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e\": container with ID starting with 606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e not found: ID does not exist" containerID="606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e"
Sep 18 16:42:57.539026 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.538988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e"} err="failed to get container status \"606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e\": rpc error: code = NotFound desc = could not find container \"606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e\": container with ID starting with 606f757deee579929e8b58f76e36346fadedba00b079fe869a1aade629f8a10e not found: ID does not exist"
Sep 18 16:42:57.539026 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.539003    2563 scope.go:117] "RemoveContainer" containerID="b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093"
Sep 18 16:42:57.539220 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:42:57.539205    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093\": container with ID starting with b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093 not found: ID does not exist" containerID="b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093"
Sep 18 16:42:57.539261 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.539224    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093"} err="failed to get container status \"b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093\": rpc error: code = NotFound desc = could not find container \"b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093\": container with ID starting with b2f127ff5ce3467827057354a90b018344fb7988149c76994d7296e7951d8093 not found: ID does not exist"
Sep 18 16:42:57.542942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.542918    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-58298-predictor-97d84ccc4-pg527"]
Sep 18 16:42:57.556976 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.556955    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"]
Sep 18 16:42:57.561704 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.561681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:57.561985 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.561970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.563928 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.563910    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 16:42:57.573080 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.573058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"]
Sep 18 16:42:57.631994 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.631909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.631994 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.631953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d976b2ee-0435-47ce-a33c-1995934f216d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.631994 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.631970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.632200 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.632075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfx5\" (UniqueName: \"kubernetes.io/projected/d976b2ee-0435-47ce-a33c-1995934f216d-kube-api-access-jkfx5\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.632200 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.632130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.733281 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.733384 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71e99af-6719-4969-a1f2-e6aebf423b15-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.733384 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.733384 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktn9s\" (UniqueName: \"kubernetes.io/projected/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-api-access-ktn9s\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.733384 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.733515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71e99af-6719-4969-a1f2-e6aebf423b15-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.733515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d976b2ee-0435-47ce-a33c-1995934f216d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.733515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.733656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.733656 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfx5\" (UniqueName: \"kubernetes.io/projected/d976b2ee-0435-47ce-a33c-1995934f216d-kube-api-access-jkfx5\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.733729 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:42:57.733669    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 16:42:57.733771 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:42:57.733742    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls podName:d976b2ee-0435-47ce-a33c-1995934f216d nodeName:}" failed. No retries permitted until 2026-09-18 16:42:58.233722292 +0000 UTC m=+1590.233983080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls") pod "isvc-canary-create-v2-predictor-667f95d746-82srz" (UID: "d976b2ee-0435-47ce-a33c-1995934f216d") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 16:42:57.733836 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.733815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d976b2ee-0435-47ce-a33c-1995934f216d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.734113 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.734093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.734160 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.734093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.743309 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.743287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfx5\" (UniqueName: \"kubernetes.io/projected/d976b2ee-0435-47ce-a33c-1995934f216d-kube-api-access-jkfx5\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:57.825301 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.825261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:42:57.834469 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.834444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktn9s\" (UniqueName: \"kubernetes.io/projected/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-api-access-ktn9s\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.834580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.834479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.834580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.834495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71e99af-6719-4969-a1f2-e6aebf423b15-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.834660 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.834622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.834760 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.834745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71e99af-6719-4969-a1f2-e6aebf423b15-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.834960 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.834937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71e99af-6719-4969-a1f2-e6aebf423b15-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.835177 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.835157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.835263 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.835230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.837259 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.837239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71e99af-6719-4969-a1f2-e6aebf423b15-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.842388 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.842367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktn9s\" (UniqueName: \"kubernetes.io/projected/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-api-access-ktn9s\") pod \"isvc-canary-create-predictor-76b74f4d6c-khx4g\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:57.872254 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:57.872236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:42:58.144993 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.144966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"]
Sep 18 16:42:58.147474 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:42:58.147446    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71e99af_6719_4969_a1f2_e6aebf423b15.slice/crio-2e8ba171020b7e293e8e52cf574022b629a6390a52581742f4d81bdabacc2744 WatchSource:0}: Error finding container 2e8ba171020b7e293e8e52cf574022b629a6390a52581742f4d81bdabacc2744: Status 404 returned error can't find the container with id 2e8ba171020b7e293e8e52cf574022b629a6390a52581742f4d81bdabacc2744
Sep 18 16:42:58.217017 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.216977    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910f5941-54c7-4a6e-bb72-24c9b100edab" path="/var/lib/kubelet/pods/910f5941-54c7-4a6e-bb72-24c9b100edab/volumes"
Sep 18 16:42:58.238063 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.238039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:58.240637 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.240609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-82srz\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:58.403941 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.403858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:42:58.519640 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.519599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerStarted","Data":"7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3"}
Sep 18 16:42:58.519844 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.519677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerStarted","Data":"2e8ba171020b7e293e8e52cf574022b629a6390a52581742f4d81bdabacc2744"}
Sep 18 16:42:58.683715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:58.683688    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"]
Sep 18 16:42:58.686154 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:42:58.686124    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd976b2ee_0435_47ce_a33c_1995934f216d.slice/crio-211cb61bba362bc5c8f734d185a27398e5bc2c12c89c78578feff9f91a685432 WatchSource:0}: Error finding container 211cb61bba362bc5c8f734d185a27398e5bc2c12c89c78578feff9f91a685432: Status 404 returned error can't find the container with id 211cb61bba362bc5c8f734d185a27398e5bc2c12c89c78578feff9f91a685432
Sep 18 16:42:59.526707 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:59.526673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerStarted","Data":"a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea"}
Sep 18 16:42:59.527064 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:59.526757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerStarted","Data":"211cb61bba362bc5c8f734d185a27398e5bc2c12c89c78578feff9f91a685432"}
Sep 18 16:42:59.852389 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:42:59.852305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:43:02.541264 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:02.541231    2563 generic.go:356] "Generic (PLEG): container finished" podID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerID="7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3" exitCode=0
Sep 18 16:43:02.541667 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:02.541305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerDied","Data":"7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3"}
Sep 18 16:43:03.547904 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.547872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerStarted","Data":"53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d"}
Sep 18 16:43:03.548354 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.547943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerStarted","Data":"a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2"}
Sep 18 16:43:03.548354 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.548297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:43:03.548480 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.548462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:43:03.549533 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.549504    2563 generic.go:356] "Generic (PLEG): container finished" podID="d976b2ee-0435-47ce-a33c-1995934f216d" containerID="a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea" exitCode=0
Sep 18 16:43:03.549689 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.549545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerDied","Data":"a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea"}
Sep 18 16:43:03.550026 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.550000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:03.569391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:03.569347    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podStartSLOduration=2.9623059449999998 podStartE2EDuration="6.569332742s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.607026797s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:03.566969527 +0000 UTC m=+1595.567230356" watchObservedRunningTime="2026-09-18 16:43:03.569332742 +0000 UTC m=+1595.569593553"
Sep 18 16:43:04.556731 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.556690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerStarted","Data":"9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898"}
Sep 18 16:43:04.557216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.556765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerStarted","Data":"26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957"}
Sep 18 16:43:04.557216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.556993    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:04.557216 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.557158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:43:04.557389 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.557322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:43:04.558600 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.558571    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:04.576977 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:04.576883    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podStartSLOduration=3.69149278 podStartE2EDuration="7.57686691s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88537413s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:04.57580905 +0000 UTC m=+1596.576069861" watchObservedRunningTime="2026-09-18 16:43:04.57686691 +0000 UTC m=+1596.577127721"
Sep 18 16:43:05.561376 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:05.561329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:05.561896 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:05.561451    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:10.566018 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:10.565819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:43:10.566793 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:10.566760    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:10.566946 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:10.566916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:43:10.567370 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:10.567350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:20.567464 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:20.567418    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:20.567908 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:20.567432    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:30.566871 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:30.566784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:30.567278 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:30.567258    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:40.567313 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:40.567269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:43:40.567795 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:40.567269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:50.567635 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:50.567598    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:43:50.567635 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:43:50.567614    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:44:00.567776 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:00.567736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:44:00.568324 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:00.567736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:44:10.567389 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:10.567352    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:44:10.567875 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:10.567350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:44:18.213258 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:18.213208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:44:20.217862 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:20.217835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:44:28.218160 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:28.218132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:44:37.562487 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.562456    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"]
Sep 18 16:44:37.563058 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.562746    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" containerID="cri-o://a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2" gracePeriod=30
Sep 18 16:44:37.563058 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.562932    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kube-rbac-proxy" containerID="cri-o://53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d" gracePeriod=30
Sep 18 16:44:37.566187 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.566164    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"]
Sep 18 16:44:37.566446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.566408    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" containerID="cri-o://26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957" gracePeriod=30
Sep 18 16:44:37.566595 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.566464    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kube-rbac-proxy" containerID="cri-o://9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898" gracePeriod=30
Sep 18 16:44:37.650121 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.650086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:37.710494 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.710465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:37.953015 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.952985    2563 generic.go:356] "Generic (PLEG): container finished" podID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerID="53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d" exitCode=2
Sep 18 16:44:37.953204 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.953053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerDied","Data":"53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d"}
Sep 18 16:44:37.955108 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.955088    2563 generic.go:356] "Generic (PLEG): container finished" podID="d976b2ee-0435-47ce-a33c-1995934f216d" containerID="9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898" exitCode=2
Sep 18 16:44:37.955208 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:37.955116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerDied","Data":"9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898"}
Sep 18 16:44:38.213593 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:38.213468    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 18 16:44:40.215491 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:40.215393    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 16:44:40.561802 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:40.561709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 18 16:44:40.561802 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:40.561714    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 16:44:42.605979 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.605954    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:44:42.635788 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.635724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71e99af-6719-4969-a1f2-e6aebf423b15-kserve-provision-location\") pod \"a71e99af-6719-4969-a1f2-e6aebf423b15\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") "
Sep 18 16:44:42.635935 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.635796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-rbac-proxy-sar-config\") pod \"a71e99af-6719-4969-a1f2-e6aebf423b15\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") "
Sep 18 16:44:42.635935 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.635867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71e99af-6719-4969-a1f2-e6aebf423b15-proxy-tls\") pod \"a71e99af-6719-4969-a1f2-e6aebf423b15\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") "
Sep 18 16:44:42.635935 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.635906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-cabundle-cert\") pod \"a71e99af-6719-4969-a1f2-e6aebf423b15\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") "
Sep 18 16:44:42.636102 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.635943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktn9s\" (UniqueName: \"kubernetes.io/projected/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-api-access-ktn9s\") pod \"a71e99af-6719-4969-a1f2-e6aebf423b15\" (UID: \"a71e99af-6719-4969-a1f2-e6aebf423b15\") "
Sep 18 16:44:42.636102 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.636063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71e99af-6719-4969-a1f2-e6aebf423b15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a71e99af-6719-4969-a1f2-e6aebf423b15" (UID: "a71e99af-6719-4969-a1f2-e6aebf423b15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:42.636226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.636138    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a71e99af-6719-4969-a1f2-e6aebf423b15-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.636226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.636171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a71e99af-6719-4969-a1f2-e6aebf423b15" (UID: "a71e99af-6719-4969-a1f2-e6aebf423b15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:44:42.636479 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.636442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a71e99af-6719-4969-a1f2-e6aebf423b15" (UID: "a71e99af-6719-4969-a1f2-e6aebf423b15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:44:42.638713 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.638687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71e99af-6719-4969-a1f2-e6aebf423b15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a71e99af-6719-4969-a1f2-e6aebf423b15" (UID: "a71e99af-6719-4969-a1f2-e6aebf423b15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:44:42.638713 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.638702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-api-access-ktn9s" (OuterVolumeSpecName: "kube-api-access-ktn9s") pod "a71e99af-6719-4969-a1f2-e6aebf423b15" (UID: "a71e99af-6719-4969-a1f2-e6aebf423b15"). InnerVolumeSpecName "kube-api-access-ktn9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:44:42.642943 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.642924    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:44:42.664318 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.664287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:42.736667 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.736637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-kube-rbac-proxy-sar-config\") pod \"d976b2ee-0435-47ce-a33c-1995934f216d\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") "
Sep 18 16:44:42.736822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.736688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d976b2ee-0435-47ce-a33c-1995934f216d-kserve-provision-location\") pod \"d976b2ee-0435-47ce-a33c-1995934f216d\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") "
Sep 18 16:44:42.736822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.736715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-cabundle-cert\") pod \"d976b2ee-0435-47ce-a33c-1995934f216d\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") "
Sep 18 16:44:42.736822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.736741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkfx5\" (UniqueName: \"kubernetes.io/projected/d976b2ee-0435-47ce-a33c-1995934f216d-kube-api-access-jkfx5\") pod \"d976b2ee-0435-47ce-a33c-1995934f216d\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") "
Sep 18 16:44:42.736822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.736801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls\") pod \"d976b2ee-0435-47ce-a33c-1995934f216d\" (UID: \"d976b2ee-0435-47ce-a33c-1995934f216d\") "
Sep 18 16:44:42.737031 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.736960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d976b2ee-0435-47ce-a33c-1995934f216d" (UID: "d976b2ee-0435-47ce-a33c-1995934f216d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:44:42.737083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737052    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.737083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737065    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a71e99af-6719-4969-a1f2-e6aebf423b15-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.737083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737075    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.737207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d976b2ee-0435-47ce-a33c-1995934f216d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d976b2ee-0435-47ce-a33c-1995934f216d" (UID: "d976b2ee-0435-47ce-a33c-1995934f216d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:42.737207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737084    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktn9s\" (UniqueName: \"kubernetes.io/projected/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-api-access-ktn9s\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.737207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d976b2ee-0435-47ce-a33c-1995934f216d" (UID: "d976b2ee-0435-47ce-a33c-1995934f216d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:44:42.737207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.737127    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a71e99af-6719-4969-a1f2-e6aebf423b15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.739066 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.738950    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d976b2ee-0435-47ce-a33c-1995934f216d-kube-api-access-jkfx5" (OuterVolumeSpecName: "kube-api-access-jkfx5") pod "d976b2ee-0435-47ce-a33c-1995934f216d" (UID: "d976b2ee-0435-47ce-a33c-1995934f216d"). InnerVolumeSpecName "kube-api-access-jkfx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:44:42.739255 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.739131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d976b2ee-0435-47ce-a33c-1995934f216d" (UID: "d976b2ee-0435-47ce-a33c-1995934f216d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:44:42.740942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.740920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:42.741817 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.741795    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"]
Sep 18 16:44:42.742113 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742096    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kube-rbac-proxy"
Sep 18 16:44:42.742113 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742112    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kube-rbac-proxy"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742122    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kube-rbac-proxy"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742130    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kube-rbac-proxy"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742140    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742146    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742159    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742164    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742177    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="storage-initializer"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742186    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="storage-initializer"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742197    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="storage-initializer"
Sep 18 16:44:42.742257 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742205    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="storage-initializer"
Sep 18 16:44:42.742717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742283    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kube-rbac-proxy"
Sep 18 16:44:42.742717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742294    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" containerName="kserve-container"
Sep 18 16:44:42.742717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742300    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kserve-container"
Sep 18 16:44:42.742717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.742307    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerName="kube-rbac-proxy"
Sep 18 16:44:42.745446 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.745431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.747449 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.747429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 16:44:42.747693 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.747679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 16:44:42.755730 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.755710    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"]
Sep 18 16:44:42.816965 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.816940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:42.838309 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2980fd2-8c99-41c1-8612-a0bef0bb2841-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.838445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgp4n\" (UniqueName: \"kubernetes.io/projected/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-api-access-xgp4n\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.838445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.838445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.838445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.838626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838481    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d976b2ee-0435-47ce-a33c-1995934f216d-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.838626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838495    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d976b2ee-0435-47ce-a33c-1995934f216d-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.838626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838505    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkfx5\" (UniqueName: \"kubernetes.io/projected/d976b2ee-0435-47ce-a33c-1995934f216d-kube-api-access-jkfx5\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.838626 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.838519    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d976b2ee-0435-47ce-a33c-1995934f216d-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:44:42.909840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.909816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:44:42.938988 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.938963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2980fd2-8c99-41c1-8612-a0bef0bb2841-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgp4n\" (UniqueName: \"kubernetes.io/projected/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-api-access-xgp4n\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939083 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939159 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939159 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939530 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.939960 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.939870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.941588 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.941555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2980fd2-8c99-41c1-8612-a0bef0bb2841-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.947437 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.947421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgp4n\" (UniqueName: \"kubernetes.io/projected/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-api-access-xgp4n\") pod \"isvc-canary-promote-predictor-6546f8697f-5kh84\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:42.982453 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.982421    2563 generic.go:356] "Generic (PLEG): container finished" podID="a71e99af-6719-4969-a1f2-e6aebf423b15" containerID="a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2" exitCode=0
Sep 18 16:44:42.982579 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.982487    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"
Sep 18 16:44:42.982579 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.982507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerDied","Data":"a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2"}
Sep 18 16:44:42.982710 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.982582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g" event={"ID":"a71e99af-6719-4969-a1f2-e6aebf423b15","Type":"ContainerDied","Data":"2e8ba171020b7e293e8e52cf574022b629a6390a52581742f4d81bdabacc2744"}
Sep 18 16:44:42.982710 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.982618    2563 scope.go:117] "RemoveContainer" containerID="53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d"
Sep 18 16:44:42.984708 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.984683    2563 generic.go:356] "Generic (PLEG): container finished" podID="d976b2ee-0435-47ce-a33c-1995934f216d" containerID="26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957" exitCode=0
Sep 18 16:44:42.984804 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.984730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"
Sep 18 16:44:42.984804 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.984778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerDied","Data":"26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957"}
Sep 18 16:44:42.984883 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.984831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz" event={"ID":"d976b2ee-0435-47ce-a33c-1995934f216d","Type":"ContainerDied","Data":"211cb61bba362bc5c8f734d185a27398e5bc2c12c89c78578feff9f91a685432"}
Sep 18 16:44:42.991895 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.991879    2563 scope.go:117] "RemoveContainer" containerID="a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2"
Sep 18 16:44:42.999958 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:42.999940    2563 scope.go:117] "RemoveContainer" containerID="7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3"
Sep 18 16:44:43.008102 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.008080    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"]
Sep 18 16:44:43.008522 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.008504    2563 scope.go:117] "RemoveContainer" containerID="53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d"
Sep 18 16:44:43.008835 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:44:43.008817    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d\": container with ID starting with 53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d not found: ID does not exist" containerID="53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d"
Sep 18 16:44:43.008912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.008845    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d"} err="failed to get container status \"53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d\": rpc error: code = NotFound desc = could not find container \"53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d\": container with ID starting with 53373adcdf2e9368f1f438323b283ba6e410d410c75d451b064d0e699efeb35d not found: ID does not exist"
Sep 18 16:44:43.008912 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.008862    2563 scope.go:117] "RemoveContainer" containerID="a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2"
Sep 18 16:44:43.009137 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:44:43.009120    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2\": container with ID starting with a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2 not found: ID does not exist" containerID="a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2"
Sep 18 16:44:43.009208 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.009145    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2"} err="failed to get container status \"a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2\": rpc error: code = NotFound desc = could not find container \"a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2\": container with ID starting with a2778d832f42b9cb8ae96bfd87074d15661cfd43f865054612071273ec07fec2 not found: ID does not exist"
Sep 18 16:44:43.009208 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.009160    2563 scope.go:117] "RemoveContainer" containerID="7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3"
Sep 18 16:44:43.009415 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:44:43.009399    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3\": container with ID starting with 7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3 not found: ID does not exist" containerID="7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3"
Sep 18 16:44:43.009456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.009420    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3"} err="failed to get container status \"7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3\": rpc error: code = NotFound desc = could not find container \"7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3\": container with ID starting with 7544b6e3c81df46d3b099ad0ae6545596e5aed5e285e1d1c77dbd05339ec3fc3 not found: ID does not exist"
Sep 18 16:44:43.009456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.009433    2563 scope.go:117] "RemoveContainer" containerID="9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898"
Sep 18 16:44:43.009862 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.009840    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-khx4g"]
Sep 18 16:44:43.018466 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.018445    2563 scope.go:117] "RemoveContainer" containerID="26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957"
Sep 18 16:44:43.020508 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.020488    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"]
Sep 18 16:44:43.025983 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.025962    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-82srz"]
Sep 18 16:44:43.026456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.026442    2563 scope.go:117] "RemoveContainer" containerID="a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea"
Sep 18 16:44:43.033645 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.033625    2563 scope.go:117] "RemoveContainer" containerID="9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898"
Sep 18 16:44:43.033947 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:44:43.033922    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898\": container with ID starting with 9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898 not found: ID does not exist" containerID="9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898"
Sep 18 16:44:43.034006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.033954    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898"} err="failed to get container status \"9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898\": rpc error: code = NotFound desc = could not find container \"9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898\": container with ID starting with 9742937fe1672fd33e9b5808101a76106f2ad6d11c8da09bae0d8a5f4a52a898 not found: ID does not exist"
Sep 18 16:44:43.034006 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.033983    2563 scope.go:117] "RemoveContainer" containerID="26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957"
Sep 18 16:44:43.034209 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:44:43.034196    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957\": container with ID starting with 26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957 not found: ID does not exist" containerID="26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957"
Sep 18 16:44:43.034276 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.034213    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957"} err="failed to get container status \"26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957\": rpc error: code = NotFound desc = could not find container \"26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957\": container with ID starting with 26499f6df6bfcc33255d4e6d932c8f2436af4527c8bfdb0385bf173ff1ce5957 not found: ID does not exist"
Sep 18 16:44:43.034276 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.034225    2563 scope.go:117] "RemoveContainer" containerID="a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea"
Sep 18 16:44:43.034430 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:44:43.034412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea\": container with ID starting with a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea not found: ID does not exist" containerID="a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea"
Sep 18 16:44:43.034467 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.034437    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea"} err="failed to get container status \"a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea\": rpc error: code = NotFound desc = could not find container \"a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea\": container with ID starting with a17e5d5bab919e6d49d4f1554083ee361f6ce78b8e9e17bbf6441422d9fee1ea not found: ID does not exist"
Sep 18 16:44:43.055894 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.055866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:43.327217 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.327193    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"]
Sep 18 16:44:43.329788 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:44:43.329762    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2980fd2_8c99_41c1_8612_a0bef0bb2841.slice/crio-4b39bd58cdcea81547258f29125fef449efc19b822b279af5371acd6cd7578eb WatchSource:0}: Error finding container 4b39bd58cdcea81547258f29125fef449efc19b822b279af5371acd6cd7578eb: Status 404 returned error can't find the container with id 4b39bd58cdcea81547258f29125fef449efc19b822b279af5371acd6cd7578eb
Sep 18 16:44:43.993080 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.993043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerStarted","Data":"b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2"}
Sep 18 16:44:43.993485 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:43.993125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerStarted","Data":"4b39bd58cdcea81547258f29125fef449efc19b822b279af5371acd6cd7578eb"}
Sep 18 16:44:44.217360 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:44.217323    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71e99af-6719-4969-a1f2-e6aebf423b15" path="/var/lib/kubelet/pods/a71e99af-6719-4969-a1f2-e6aebf423b15/volumes"
Sep 18 16:44:44.217840 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:44.217827    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d976b2ee-0435-47ce-a33c-1995934f216d" path="/var/lib/kubelet/pods/d976b2ee-0435-47ce-a33c-1995934f216d/volumes"
Sep 18 16:44:48.011520 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:48.011490    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerID="b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2" exitCode=0
Sep 18 16:44:48.011933 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:48.011587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerDied","Data":"b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2"}
Sep 18 16:44:49.018492 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:49.018458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerStarted","Data":"975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215"}
Sep 18 16:44:49.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:49.018521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerStarted","Data":"6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543"}
Sep 18 16:44:49.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:49.018831    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:49.018973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:49.018969    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:49.020258 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:49.020238    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:44:49.041694 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:49.041653    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podStartSLOduration=3.034046944 podStartE2EDuration="7.041640168s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.007593224s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:49.041143653 +0000 UTC m=+1701.041404488" watchObservedRunningTime="2026-09-18 16:44:49.041640168 +0000 UTC m=+1701.041900979"
Sep 18 16:44:50.029906 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:50.029866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:44:55.034973 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:55.034941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:44:55.035467 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:44:55.035439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:05.035456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:05.035406    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:15.036029 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:15.035991    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:25.036050 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:25.036005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:30.278773 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:30.278743    2563 scope.go:117] "RemoveContainer" containerID="584a4571b9e8fa027e65d794a0eeab6cc50e4eaf63348ffd7eb6dc5ade3a57fc"
Sep 18 16:45:35.035789 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:35.035740    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:45.035787 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:45.035746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:55.036365 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:55.036321    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:45:59.212868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:45:59.212826    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 18 16:46:05.508737 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:05.508697    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"]
Sep 18 16:46:05.509172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:05.508980    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container" containerID="cri-o://6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543" gracePeriod=30
Sep 18 16:46:05.509172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:05.509040    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kube-rbac-proxy" containerID="cri-o://975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215" gracePeriod=30
Sep 18 16:46:05.509787 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:05.509709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:46:06.344194 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:06.344162    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerID="975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215" exitCode=2
Sep 18 16:46:06.344366 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:06.344229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerDied","Data":"975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215"}
Sep 18 16:46:10.030541 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.030448    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 18 16:46:10.345445 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.345414    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:46:10.361158 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.361131    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerID="6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543" exitCode=0
Sep 18 16:46:10.361297 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.361196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerDied","Data":"6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543"}
Sep 18 16:46:10.361297 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.361253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84" event={"ID":"f2980fd2-8c99-41c1-8612-a0bef0bb2841","Type":"ContainerDied","Data":"4b39bd58cdcea81547258f29125fef449efc19b822b279af5371acd6cd7578eb"}
Sep 18 16:46:10.361297 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.361264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"
Sep 18 16:46:10.361297 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.361281    2563 scope.go:117] "RemoveContainer" containerID="975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215"
Sep 18 16:46:10.370665 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.370644    2563 scope.go:117] "RemoveContainer" containerID="6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543"
Sep 18 16:46:10.378611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.378591    2563 scope.go:117] "RemoveContainer" containerID="b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2"
Sep 18 16:46:10.385915 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.385895    2563 scope.go:117] "RemoveContainer" containerID="975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215"
Sep 18 16:46:10.386179 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:46:10.386161    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215\": container with ID starting with 975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215 not found: ID does not exist" containerID="975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215"
Sep 18 16:46:10.386240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.386190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215"} err="failed to get container status \"975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215\": rpc error: code = NotFound desc = could not find container \"975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215\": container with ID starting with 975a3f34800a1e1a29c0e1c6e06dc5db60a064795a684925e0409b2a7a91e215 not found: ID does not exist"
Sep 18 16:46:10.386240 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.386207    2563 scope.go:117] "RemoveContainer" containerID="6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543"
Sep 18 16:46:10.386502 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:46:10.386480    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543\": container with ID starting with 6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543 not found: ID does not exist" containerID="6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543"
Sep 18 16:46:10.386614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.386506    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543"} err="failed to get container status \"6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543\": rpc error: code = NotFound desc = could not find container \"6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543\": container with ID starting with 6c8751590543ecf68b2f1a1cf5fd8320e2a111870920b1a4d7a784fe12134543 not found: ID does not exist"
Sep 18 16:46:10.386614 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.386522    2563 scope.go:117] "RemoveContainer" containerID="b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2"
Sep 18 16:46:10.386767 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:46:10.386751    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2\": container with ID starting with b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2 not found: ID does not exist" containerID="b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2"
Sep 18 16:46:10.386808 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.386772    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2"} err="failed to get container status \"b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2\": rpc error: code = NotFound desc = could not find container \"b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2\": container with ID starting with b4be4ed055be3e35f32b6ff14c54d625d8123b7808973b85ec177d9f106b44e2 not found: ID does not exist"
Sep 18 16:46:10.492392 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kserve-provision-location\") pod \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") "
Sep 18 16:46:10.492392 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2980fd2-8c99-41c1-8612-a0bef0bb2841-proxy-tls\") pod \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") "
Sep 18 16:46:10.492649 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492450    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-cabundle-cert\") pod \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") "
Sep 18 16:46:10.492649 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgp4n\" (UniqueName: \"kubernetes.io/projected/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-api-access-xgp4n\") pod \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") "
Sep 18 16:46:10.492649 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-rbac-proxy-sar-config\") pod \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\" (UID: \"f2980fd2-8c99-41c1-8612-a0bef0bb2841\") "
Sep 18 16:46:10.492820 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2980fd2-8c99-41c1-8612-a0bef0bb2841" (UID: "f2980fd2-8c99-41c1-8612-a0bef0bb2841"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:10.492879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2980fd2-8c99-41c1-8612-a0bef0bb2841" (UID: "f2980fd2-8c99-41c1-8612-a0bef0bb2841"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:46:10.492961 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.492924    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2980fd2-8c99-41c1-8612-a0bef0bb2841" (UID: "f2980fd2-8c99-41c1-8612-a0bef0bb2841"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:46:10.494698 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.494681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2980fd2-8c99-41c1-8612-a0bef0bb2841-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2980fd2-8c99-41c1-8612-a0bef0bb2841" (UID: "f2980fd2-8c99-41c1-8612-a0bef0bb2841"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:46:10.494905 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.494887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-api-access-xgp4n" (OuterVolumeSpecName: "kube-api-access-xgp4n") pod "f2980fd2-8c99-41c1-8612-a0bef0bb2841" (UID: "f2980fd2-8c99-41c1-8612-a0bef0bb2841"). InnerVolumeSpecName "kube-api-access-xgp4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:46:10.593072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.592995    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:46:10.593072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.593017    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2980fd2-8c99-41c1-8612-a0bef0bb2841-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:46:10.593072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.593028    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:46:10.593072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.593037    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgp4n\" (UniqueName: \"kubernetes.io/projected/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-api-access-xgp4n\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:46:10.593072 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.593045    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2980fd2-8c99-41c1-8612-a0bef0bb2841-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:46:10.690621 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.690583    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"]
Sep 18 16:46:10.692762 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:10.692738    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-5kh84"]
Sep 18 16:46:11.099660 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:11.099622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:11.203021 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:11.202982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:12.217298 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.217269    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" path="/var/lib/kubelet/pods/f2980fd2-8c99-41c1-8612-a0bef0bb2841/volumes"
Sep 18 16:46:12.867993 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.867962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:46:12.914242 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914214    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"]
Sep 18 16:46:12.914534 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914522    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="storage-initializer"
Sep 18 16:46:12.914611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="storage-initializer"
Sep 18 16:46:12.914611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container"
Sep 18 16:46:12.914611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914552    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container"
Sep 18 16:46:12.914611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914575    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kube-rbac-proxy"
Sep 18 16:46:12.914611 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914581    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kube-rbac-proxy"
Sep 18 16:46:12.914772 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914635    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kserve-container"
Sep 18 16:46:12.914772 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.914642    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2980fd2-8c99-41c1-8612-a0bef0bb2841" containerName="kube-rbac-proxy"
Sep 18 16:46:12.919100 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.919083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:12.921525 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.921504    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 16:46:12.921525 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.921514    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:46:12.921722 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.921520    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:46:12.921722 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.921695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 16:46:12.921722 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.921708    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:46:12.922538 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.922521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:46:12.922646 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.922600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:46:12.927345 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:12.927320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"]
Sep 18 16:46:13.114888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.114853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.114888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.114894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.115097 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.114970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06458284-0347-4d9c-84bb-8a0e529f9d1b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.115097 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.115021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qc2w\" (UniqueName: \"kubernetes.io/projected/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-api-access-4qc2w\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.115097 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.115043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06458284-0347-4d9c-84bb-8a0e529f9d1b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.215939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.215904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06458284-0347-4d9c-84bb-8a0e529f9d1b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.215939 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.215939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qc2w\" (UniqueName: \"kubernetes.io/projected/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-api-access-4qc2w\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.216141 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.215957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06458284-0347-4d9c-84bb-8a0e529f9d1b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.216141 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.215989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.216141 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.216011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.216387 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.216365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06458284-0347-4d9c-84bb-8a0e529f9d1b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.216764 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.216743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.216853 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.216751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.218530 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.218512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06458284-0347-4d9c-84bb-8a0e529f9d1b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.224700 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.224677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qc2w\" (UniqueName: \"kubernetes.io/projected/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-api-access-4qc2w\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.229293 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.229275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:13.522128 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:13.522044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"]
Sep 18 16:46:13.526392 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:46:13.526360    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06458284_0347_4d9c_84bb_8a0e529f9d1b.slice/crio-f618df05d0a6c7fa825da98e78dc3e9dbe09af574be534330de54e4016f0664d WatchSource:0}: Error finding container f618df05d0a6c7fa825da98e78dc3e9dbe09af574be534330de54e4016f0664d: Status 404 returned error can't find the container with id f618df05d0a6c7fa825da98e78dc3e9dbe09af574be534330de54e4016f0664d
Sep 18 16:46:14.379620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:14.379584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerStarted","Data":"b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821"}
Sep 18 16:46:14.380012 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:14.379644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerStarted","Data":"f618df05d0a6c7fa825da98e78dc3e9dbe09af574be534330de54e4016f0664d"}
Sep 18 16:46:17.393776 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:17.393738    2563 generic.go:356] "Generic (PLEG): container finished" podID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerID="b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821" exitCode=0
Sep 18 16:46:17.394157 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:17.393801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerDied","Data":"b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821"}
Sep 18 16:46:18.399634 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:18.399598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerStarted","Data":"67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf"}
Sep 18 16:46:18.400033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:18.399663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerStarted","Data":"eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270"}
Sep 18 16:46:18.400033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:18.399853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:18.421533 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:18.421489    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podStartSLOduration=2.722700132 podStartE2EDuration="6.421473958s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698773826s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:46:18.419994181 +0000 UTC m=+1790.420255006" watchObservedRunningTime="2026-09-18 16:46:18.421473958 +0000 UTC m=+1790.421734769"
Sep 18 16:46:19.404414 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:19.404378    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:19.405534 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:19.405508    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:46:20.408738 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:20.408699    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:46:25.413722 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:25.413686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:46:25.414366 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:25.414335    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:46:30.210620 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:30.210585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:46:30.212427 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:30.212400    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:46:35.415054 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:35.415013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:46:45.415235 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:45.415195    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:46:55.415159 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:46:55.415120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:47:05.414311 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:05.414269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:47:15.414764 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:15.414719    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:47:25.414366 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:25.414322    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:47:35.415702 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:35.415672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:47:43.061070 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:43.060994    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"]
Sep 18 16:47:43.061869 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:43.061242    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" containerID="cri-o://eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270" gracePeriod=30
Sep 18 16:47:43.061869 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:43.061306    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kube-rbac-proxy" containerID="cri-o://67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf" gracePeriod=30
Sep 18 16:47:43.108105 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:43.108077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:43.739822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:43.739788    2563 generic.go:356] "Generic (PLEG): container finished" podID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerID="67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf" exitCode=2
Sep 18 16:47:43.739998 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:43.739865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerDied","Data":"67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf"}
Sep 18 16:47:45.409889 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:45.409841    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 18 16:47:45.415051 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:45.415025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 18 16:47:48.010234 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.010208    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:47:48.096716 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.096630    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06458284-0347-4d9c-84bb-8a0e529f9d1b-proxy-tls\") pod \"06458284-0347-4d9c-84bb-8a0e529f9d1b\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") "
Sep 18 16:47:48.096898 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.096726    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-rbac-proxy-sar-config\") pod \"06458284-0347-4d9c-84bb-8a0e529f9d1b\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") "
Sep 18 16:47:48.096898 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.096767    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-cabundle-cert\") pod \"06458284-0347-4d9c-84bb-8a0e529f9d1b\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") "
Sep 18 16:47:48.096898 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.096783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06458284-0347-4d9c-84bb-8a0e529f9d1b-kserve-provision-location\") pod \"06458284-0347-4d9c-84bb-8a0e529f9d1b\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") "
Sep 18 16:47:48.096898 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.096809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qc2w\" (UniqueName: \"kubernetes.io/projected/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-api-access-4qc2w\") pod \"06458284-0347-4d9c-84bb-8a0e529f9d1b\" (UID: \"06458284-0347-4d9c-84bb-8a0e529f9d1b\") "
Sep 18 16:47:48.097159 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.097125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06458284-0347-4d9c-84bb-8a0e529f9d1b" (UID: "06458284-0347-4d9c-84bb-8a0e529f9d1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:47:48.097207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.097151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06458284-0347-4d9c-84bb-8a0e529f9d1b" (UID: "06458284-0347-4d9c-84bb-8a0e529f9d1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:47:48.097207 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.097131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06458284-0347-4d9c-84bb-8a0e529f9d1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06458284-0347-4d9c-84bb-8a0e529f9d1b" (UID: "06458284-0347-4d9c-84bb-8a0e529f9d1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:48.099022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.098999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06458284-0347-4d9c-84bb-8a0e529f9d1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06458284-0347-4d9c-84bb-8a0e529f9d1b" (UID: "06458284-0347-4d9c-84bb-8a0e529f9d1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:47:48.099109 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.099059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-api-access-4qc2w" (OuterVolumeSpecName: "kube-api-access-4qc2w") pod "06458284-0347-4d9c-84bb-8a0e529f9d1b" (UID: "06458284-0347-4d9c-84bb-8a0e529f9d1b"). InnerVolumeSpecName "kube-api-access-4qc2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:48.197991 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.197957    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qc2w\" (UniqueName: \"kubernetes.io/projected/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-api-access-4qc2w\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.197991 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.197987    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06458284-0347-4d9c-84bb-8a0e529f9d1b-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.197991 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.197998    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.198226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.198006    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06458284-0347-4d9c-84bb-8a0e529f9d1b-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.198226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.198015    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06458284-0347-4d9c-84bb-8a0e529f9d1b-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:47:48.765162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.765131    2563 generic.go:356] "Generic (PLEG): container finished" podID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerID="eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270" exitCode=0
Sep 18 16:47:48.765340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.765183    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"
Sep 18 16:47:48.765340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.765193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerDied","Data":"eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270"}
Sep 18 16:47:48.765340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.765242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt" event={"ID":"06458284-0347-4d9c-84bb-8a0e529f9d1b","Type":"ContainerDied","Data":"f618df05d0a6c7fa825da98e78dc3e9dbe09af574be534330de54e4016f0664d"}
Sep 18 16:47:48.765340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.765267    2563 scope.go:117] "RemoveContainer" containerID="67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf"
Sep 18 16:47:48.773583 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.773541    2563 scope.go:117] "RemoveContainer" containerID="eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270"
Sep 18 16:47:48.780800 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.780783    2563 scope.go:117] "RemoveContainer" containerID="b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821"
Sep 18 16:47:48.783317 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.783292    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"]
Sep 18 16:47:48.787376 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.787354    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-fkvgt"]
Sep 18 16:47:48.789388 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.789370    2563 scope.go:117] "RemoveContainer" containerID="67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf"
Sep 18 16:47:48.789684 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:47:48.789663    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf\": container with ID starting with 67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf not found: ID does not exist" containerID="67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf"
Sep 18 16:47:48.789773 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.789696    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf"} err="failed to get container status \"67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf\": rpc error: code = NotFound desc = could not find container \"67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf\": container with ID starting with 67df5aedac3176391a1121e95898d7e0184659f9f8451ec1e697653208747edf not found: ID does not exist"
Sep 18 16:47:48.789773 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.789721    2563 scope.go:117] "RemoveContainer" containerID="eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270"
Sep 18 16:47:48.789976 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:47:48.789956    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270\": container with ID starting with eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270 not found: ID does not exist" containerID="eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270"
Sep 18 16:47:48.790022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.789986    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270"} err="failed to get container status \"eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270\": rpc error: code = NotFound desc = could not find container \"eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270\": container with ID starting with eb6a2bb41e078d476038f375bc2859113c9395c7e7b42883f4aceb3f74b38270 not found: ID does not exist"
Sep 18 16:47:48.790022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.790009    2563 scope.go:117] "RemoveContainer" containerID="b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821"
Sep 18 16:47:48.790226 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:47:48.790211    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821\": container with ID starting with b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821 not found: ID does not exist" containerID="b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821"
Sep 18 16:47:48.790287 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:48.790234    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821"} err="failed to get container status \"b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821\": rpc error: code = NotFound desc = could not find container \"b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821\": container with ID starting with b81505d12d506410e7631746101a0aa17c16591be3b4dc7439c37317b13f3821 not found: ID does not exist"
Sep 18 16:47:50.217515 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:50.217472    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" path="/var/lib/kubelet/pods/06458284-0347-4d9c-84bb-8a0e529f9d1b/volumes"
Sep 18 16:47:57.516296 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.516251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:57.557379 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557346    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"]
Sep 18 16:47:57.557720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557705    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="storage-initializer"
Sep 18 16:47:57.557720 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557721    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="storage-initializer"
Sep 18 16:47:57.557850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557729    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kube-rbac-proxy"
Sep 18 16:47:57.557850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557735    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kube-rbac-proxy"
Sep 18 16:47:57.557850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container"
Sep 18 16:47:57.557850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container"
Sep 18 16:47:57.557850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557803    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kserve-container"
Sep 18 16:47:57.557850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.557812    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="06458284-0347-4d9c-84bb-8a0e529f9d1b" containerName="kube-rbac-proxy"
Sep 18 16:47:57.560990 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.560972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.562953 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.562933    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 16:47:57.563209 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.563188    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 18 16:47:57.563290 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.563262    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 16:47:57.563814 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.563801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 16:47:57.563888 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.563799    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6lh7v\""
Sep 18 16:47:57.570853 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.570832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"]
Sep 18 16:47:57.680037 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.680007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-api-access-l6bzl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.680200 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.680062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.680200 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.680083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/132b9401-18e2-4ed8-afef-702ab650d6ba-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.708172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.708147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:47:57.780668 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.780594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-api-access-l6bzl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.780668 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.780647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.780831 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.780671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/132b9401-18e2-4ed8-afef-702ab650d6ba-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.781350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.781291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.783162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.783132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/132b9401-18e2-4ed8-afef-702ab650d6ba-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.789979 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.789959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-api-access-l6bzl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-zmpql\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:57.871548 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:57.871508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:58.140265 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:58.140225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"]
Sep 18 16:47:58.143080 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:47:58.143053    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132b9401_18e2_4ed8_afef_702ab650d6ba.slice/crio-ffd335c48f6674ecad6d39205e3547480f8f57157fdb31cd79efe82a830d2dd6 WatchSource:0}: Error finding container ffd335c48f6674ecad6d39205e3547480f8f57157fdb31cd79efe82a830d2dd6: Status 404 returned error can't find the container with id ffd335c48f6674ecad6d39205e3547480f8f57157fdb31cd79efe82a830d2dd6
Sep 18 16:47:58.144975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:58.144959    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:47:58.811552 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:58.811513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" event={"ID":"132b9401-18e2-4ed8-afef-702ab650d6ba","Type":"ContainerStarted","Data":"ffd335c48f6674ecad6d39205e3547480f8f57157fdb31cd79efe82a830d2dd6"}
Sep 18 16:47:59.817329 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:59.817243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" event={"ID":"132b9401-18e2-4ed8-afef-702ab650d6ba","Type":"ContainerStarted","Data":"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"}
Sep 18 16:47:59.817329 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:59.817306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:59.817329 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:59.817328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" event={"ID":"132b9401-18e2-4ed8-afef-702ab650d6ba","Type":"ContainerStarted","Data":"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"}
Sep 18 16:47:59.817792 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:59.817383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:59.819914 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:59.819894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:47:59.835878 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:47:59.835828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" podStartSLOduration=1.435821299 podStartE2EDuration="2.835814187s" totalImagesPullingTime="1.399992888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:59.834119526 +0000 UTC m=+1891.834380348" watchObservedRunningTime="2026-09-18 16:47:59.835814187 +0000 UTC m=+1891.836074998"
Sep 18 16:48:06.829076 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:06.829045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:48:07.552132 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.552100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:07.600964 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.600926    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"]
Sep 18 16:48:07.604680 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.604659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.606688 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.606666    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 16:48:07.606796 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.606719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 16:48:07.606850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.606796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 18 16:48:07.606975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.606957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:48:07.613590 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.613536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"]
Sep 18 16:48:07.651389 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.651360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mss5\" (UniqueName: \"kubernetes.io/projected/a692a891-8165-486f-b355-6a53917b00e2-kube-api-access-7mss5\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.651485 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.651405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.651485 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.651459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692a891-8165-486f-b355-6a53917b00e2-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.651610 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.651492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.651610 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.651536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.751961 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.751927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692a891-8165-486f-b355-6a53917b00e2-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752105 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.751972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752105 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.751998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752105 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.752051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mss5\" (UniqueName: \"kubernetes.io/projected/a692a891-8165-486f-b355-6a53917b00e2-kube-api-access-7mss5\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752105 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:48:07.752090    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 18 16:48:07.752342 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:48:07.752156    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls podName:a692a891-8165-486f-b355-6a53917b00e2 nodeName:}" failed. No retries permitted until 2026-09-18 16:48:08.252134731 +0000 UTC m=+1900.252395523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls") pod "isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" (UID: "a692a891-8165-486f-b355-6a53917b00e2") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 18 16:48:07.752342 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.752091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752494 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.752366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692a891-8165-486f-b355-6a53917b00e2-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752673 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.752655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.752831 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.752813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:07.762199 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:07.762178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mss5\" (UniqueName: \"kubernetes.io/projected/a692a891-8165-486f-b355-6a53917b00e2-kube-api-access-7mss5\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:08.112964 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:08.112934    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:08.254667 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:08.254630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:08.257391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:08.257357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-kqp6p\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:08.515662 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:08.515626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:08.790476 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:08.790390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"]
Sep 18 16:48:08.793633 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:48:08.793606    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda692a891_8165_486f_b355_6a53917b00e2.slice/crio-7ef633ca5438ec64722223bbd5776e8c61ab62b800bd91a9f03a3be938da0edc WatchSource:0}: Error finding container 7ef633ca5438ec64722223bbd5776e8c61ab62b800bd91a9f03a3be938da0edc: Status 404 returned error can't find the container with id 7ef633ca5438ec64722223bbd5776e8c61ab62b800bd91a9f03a3be938da0edc
Sep 18 16:48:08.852938 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:08.852911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerStarted","Data":"7ef633ca5438ec64722223bbd5776e8c61ab62b800bd91a9f03a3be938da0edc"}
Sep 18 16:48:09.858583 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:09.858536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerStarted","Data":"2e186ffbba41be29eb14d1d957820dd41932f9d5139659cd2c520eeb7c7bac21"}
Sep 18 16:48:12.871856 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:12.871769    2563 generic.go:356] "Generic (PLEG): container finished" podID="a692a891-8165-486f-b355-6a53917b00e2" containerID="2e186ffbba41be29eb14d1d957820dd41932f9d5139659cd2c520eeb7c7bac21" exitCode=0
Sep 18 16:48:12.872210 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:12.871846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerDied","Data":"2e186ffbba41be29eb14d1d957820dd41932f9d5139659cd2c520eeb7c7bac21"}
Sep 18 16:48:13.878897 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.878867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerStarted","Data":"897bd5eefea2267b14c84a4279eb2286eb23e66dbf489ffd470ed3bd2d8c80b5"}
Sep 18 16:48:13.879391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.878933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerStarted","Data":"7f00880978f3b690196617b77a8eb4429b8ca8c5c3abab001961ce7efeb3fbbe"}
Sep 18 16:48:13.879391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.878959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerStarted","Data":"2b2d7386e8b105f80319d5bfba525af8ef8346d725014a5cb0365ac60cd186b0"}
Sep 18 16:48:13.879391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.879284    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:13.879391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.879346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:13.879391 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.879371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:13.881277 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.881249    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:13.882037 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.882012    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:48:13.901683 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:13.901637    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podStartSLOduration=3.21147952 podStartE2EDuration="6.901625144s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690145624s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:48:13.900728164 +0000 UTC m=+1905.900988995" watchObservedRunningTime="2026-09-18 16:48:13.901625144 +0000 UTC m=+1905.901885952"
Sep 18 16:48:14.883327 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:14.883280    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:14.883781 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:14.883692    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:48:19.888388 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:19.888358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:48:19.888923 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:19.888890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:19.889259 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:19.889238    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:48:29.889788 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:29.889734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:29.890298 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:29.890271    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:48:35.918295 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:35.918254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:48:39.889059 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:39.889014    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:39.889541 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:39.889499    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:48:49.889468 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:49.889424    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:49.889969 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:49.889941    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:48:59.889409 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:59.889369    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:48:59.891872 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:48:59.889853    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:49:09.889112 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:09.888990    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:49:09.889576 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:09.889541    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:49:13.897150 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:13.897106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:19.889554 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:19.889505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:49:19.890141 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:19.889998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:49:29.890307 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:29.890269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:49:29.890721 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:29.890533    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:49:43.037970 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.037935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-zmpql_132b9401-18e2-4ed8-afef-702ab650d6ba/kserve-container/0.log"
Sep 18 16:49:43.245020 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.244987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"]
Sep 18 16:49:43.245435 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.245301    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" containerID="cri-o://2b2d7386e8b105f80319d5bfba525af8ef8346d725014a5cb0365ac60cd186b0" gracePeriod=30
Sep 18 16:49:43.245585 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.245393    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" containerID="cri-o://897bd5eefea2267b14c84a4279eb2286eb23e66dbf489ffd470ed3bd2d8c80b5" gracePeriod=30
Sep 18 16:49:43.245687 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.245378    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" containerID="cri-o://7f00880978f3b690196617b77a8eb4429b8ca8c5c3abab001961ce7efeb3fbbe" gracePeriod=30
Sep 18 16:49:43.264300 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.264276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:43.272775 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.272748    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"]
Sep 18 16:49:43.272962 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.272943    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kserve-container" containerID="cri-o://c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f" gracePeriod=30
Sep 18 16:49:43.273046 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.273024    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kube-rbac-proxy" containerID="cri-o://2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1" gracePeriod=30
Sep 18 16:49:43.325057 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.325028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:43.523389 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.523365    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:49:43.625868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.625774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-api-access-l6bzl\") pod \"132b9401-18e2-4ed8-afef-702ab650d6ba\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") "
Sep 18 16:49:43.625868 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.625867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/132b9401-18e2-4ed8-afef-702ab650d6ba-proxy-tls\") pod \"132b9401-18e2-4ed8-afef-702ab650d6ba\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") "
Sep 18 16:49:43.626062 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.625901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-rbac-proxy-sar-config\") pod \"132b9401-18e2-4ed8-afef-702ab650d6ba\" (UID: \"132b9401-18e2-4ed8-afef-702ab650d6ba\") "
Sep 18 16:49:43.626276 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.626257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "132b9401-18e2-4ed8-afef-702ab650d6ba" (UID: "132b9401-18e2-4ed8-afef-702ab650d6ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:49:43.628084 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.628059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132b9401-18e2-4ed8-afef-702ab650d6ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "132b9401-18e2-4ed8-afef-702ab650d6ba" (UID: "132b9401-18e2-4ed8-afef-702ab650d6ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:43.628173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.628156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-api-access-l6bzl" (OuterVolumeSpecName: "kube-api-access-l6bzl") pod "132b9401-18e2-4ed8-afef-702ab650d6ba" (UID: "132b9401-18e2-4ed8-afef-702ab650d6ba"). InnerVolumeSpecName "kube-api-access-l6bzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:49:43.727170 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.727125    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/132b9401-18e2-4ed8-afef-702ab650d6ba-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:49:43.727170 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.727167    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:49:43.727170 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:43.727178    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6bzl\" (UniqueName: \"kubernetes.io/projected/132b9401-18e2-4ed8-afef-702ab650d6ba-kube-api-access-l6bzl\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:49:44.245201 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.245164    2563 generic.go:356] "Generic (PLEG): container finished" podID="a692a891-8165-486f-b355-6a53917b00e2" containerID="7f00880978f3b690196617b77a8eb4429b8ca8c5c3abab001961ce7efeb3fbbe" exitCode=2
Sep 18 16:49:44.245580 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.245236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerDied","Data":"7f00880978f3b690196617b77a8eb4429b8ca8c5c3abab001961ce7efeb3fbbe"}
Sep 18 16:49:44.246793 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246770    2563 generic.go:356] "Generic (PLEG): container finished" podID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerID="2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1" exitCode=2
Sep 18 16:49:44.246793 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246789    2563 generic.go:356] "Generic (PLEG): container finished" podID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerID="c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f" exitCode=2
Sep 18 16:49:44.246942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"
Sep 18 16:49:44.246942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" event={"ID":"132b9401-18e2-4ed8-afef-702ab650d6ba","Type":"ContainerDied","Data":"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"}
Sep 18 16:49:44.246942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" event={"ID":"132b9401-18e2-4ed8-afef-702ab650d6ba","Type":"ContainerDied","Data":"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"}
Sep 18 16:49:44.246942 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql" event={"ID":"132b9401-18e2-4ed8-afef-702ab650d6ba","Type":"ContainerDied","Data":"ffd335c48f6674ecad6d39205e3547480f8f57157fdb31cd79efe82a830d2dd6"}
Sep 18 16:49:44.247111 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.246950    2563 scope.go:117] "RemoveContainer" containerID="2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"
Sep 18 16:49:44.255172 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.255152    2563 scope.go:117] "RemoveContainer" containerID="c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"
Sep 18 16:49:44.262916 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.262894    2563 scope.go:117] "RemoveContainer" containerID="2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"
Sep 18 16:49:44.263219 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:49:44.263193    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1\": container with ID starting with 2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1 not found: ID does not exist" containerID="2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"
Sep 18 16:49:44.263279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263234    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"} err="failed to get container status \"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1\": rpc error: code = NotFound desc = could not find container \"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1\": container with ID starting with 2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1 not found: ID does not exist"
Sep 18 16:49:44.263279 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263257    2563 scope.go:117] "RemoveContainer" containerID="c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"
Sep 18 16:49:44.263667 ip-10-0-136-96 kubenswrapper[2563]: E0918 16:49:44.263549    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f\": container with ID starting with c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f not found: ID does not exist" containerID="c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"
Sep 18 16:49:44.263717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263675    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"} err="failed to get container status \"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f\": rpc error: code = NotFound desc = could not find container \"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f\": container with ID starting with c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f not found: ID does not exist"
Sep 18 16:49:44.263717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263691    2563 scope.go:117] "RemoveContainer" containerID="2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"
Sep 18 16:49:44.263921 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263902    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"]
Sep 18 16:49:44.263975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263942    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1"} err="failed to get container status \"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1\": rpc error: code = NotFound desc = could not find container \"2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1\": container with ID starting with 2948f7657cf756f41e54da505fd14d37d404021996ae9987395cf133c582d1c1 not found: ID does not exist"
Sep 18 16:49:44.263975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.263956    2563 scope.go:117] "RemoveContainer" containerID="c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"
Sep 18 16:49:44.264179 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.264163    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f"} err="failed to get container status \"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f\": rpc error: code = NotFound desc = could not find container \"c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f\": container with ID starting with c2dfab9102c58d8eb7c1def9f66d10cd09fb9f2773487f72e291550dddb6cf7f not found: ID does not exist"
Sep 18 16:49:44.269959 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.269940    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-zmpql"]
Sep 18 16:49:44.884010 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.883962    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 16:49:44.903061 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:44.903030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:45.001022 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:45.000990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:49:46.221908 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:46.218298    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" path="/var/lib/kubelet/pods/132b9401-18e2-4ed8-afef-702ab650d6ba/volumes"
Sep 18 16:49:48.269830 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:48.269746    2563 generic.go:356] "Generic (PLEG): container finished" podID="a692a891-8165-486f-b355-6a53917b00e2" containerID="2b2d7386e8b105f80319d5bfba525af8ef8346d725014a5cb0365ac60cd186b0" exitCode=0
Sep 18 16:49:48.269830 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:48.269814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerDied","Data":"2b2d7386e8b105f80319d5bfba525af8ef8346d725014a5cb0365ac60cd186b0"}
Sep 18 16:49:49.883601 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:49.883534    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 16:49:49.888927 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:49.888895    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:49:49.889244 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:49.889219    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:49:54.884323 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:54.884281    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 16:49:54.884799 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:54.884434    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:49:59.884039 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:59.883996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 16:49:59.889388 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:59.889362    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:49:59.889732 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:49:59.889712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:50:00.116518 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:00.116493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:00.138981 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:00.138940    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:00.151586 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:00.151521    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:00.161984 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:00.161959    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:00.981749 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:00.981695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:00.993357 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:00.993327    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:01.003624 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:01.003547    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:01.013786 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:01.013737    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:01.854364 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:01.854338    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:01.865596 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:01.865552    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:01.876110 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:01.876082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:01.886173 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:01.886140    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:02.706047 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:02.705981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:02.717901 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:02.717860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:02.728652 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:02.728624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:02.739014 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:02.738990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:03.536142 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:03.536114    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:03.549071 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:03.549045    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:03.560275 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:03.560254    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:03.570070 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:03.570049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:04.380874 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:04.380844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:04.392249 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:04.392218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:04.402900 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:04.402877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:04.413726 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:04.413704    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:04.883987 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:04.883936    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 16:50:05.227701 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:05.227670    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:05.239659 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:05.239638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:05.249779 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:05.249759    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:05.259284 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:05.259260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:06.055923 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.055887    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:06.067882 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.067850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:06.077685 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.077653    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:06.088014 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.087990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:06.891959 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.891912    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:06.902607 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.902585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:06.912721 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.912701    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:06.923748 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:06.923727    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:07.739178 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:07.739150    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:07.751605 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:07.751578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:07.761877 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:07.761854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:07.773829 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:07.773801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:08.588478 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:08.588449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:08.599842 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:08.599815    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:08.609822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:08.609801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:08.619850 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:08.619830    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:09.421816 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.421790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:09.433965 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.433945    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:09.444063 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.444040    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:09.454550 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.454528    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:09.884226 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.884141    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 18 16:50:09.888865 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.888829    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 18 16:50:09.888987 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.888977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:50:09.889221 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.889191    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:50:09.889342 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:09.889329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:50:10.275162 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:10.275134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:10.286449 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:10.286425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:10.297163 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:10.297139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:10.307280 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:10.307255    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:11.138433 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:11.138405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:11.150678 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:11.150653    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:11.163536 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:11.163510    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:11.174488 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:11.174473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:12.059411 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:12.059377    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kserve-container/0.log"
Sep 18 16:50:12.071192 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:12.071170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/kube-rbac-proxy/0.log"
Sep 18 16:50:12.081319 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:12.081303    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/agent/0.log"
Sep 18 16:50:12.091319 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:12.091299    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-kqp6p_a692a891-8165-486f-b355-6a53917b00e2/storage-initializer/0.log"
Sep 18 16:50:13.384009 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.383969    2563 generic.go:356] "Generic (PLEG): container finished" podID="a692a891-8165-486f-b355-6a53917b00e2" containerID="897bd5eefea2267b14c84a4279eb2286eb23e66dbf489ffd470ed3bd2d8c80b5" exitCode=0
Sep 18 16:50:13.384319 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.384022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerDied","Data":"897bd5eefea2267b14c84a4279eb2286eb23e66dbf489ffd470ed3bd2d8c80b5"}
Sep 18 16:50:13.402879 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.402853    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:50:13.574975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.574856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls\") pod \"a692a891-8165-486f-b355-6a53917b00e2\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") "
Sep 18 16:50:13.574975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.574924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692a891-8165-486f-b355-6a53917b00e2-kserve-provision-location\") pod \"a692a891-8165-486f-b355-6a53917b00e2\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") "
Sep 18 16:50:13.574975 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.574967    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mss5\" (UniqueName: \"kubernetes.io/projected/a692a891-8165-486f-b355-6a53917b00e2-kube-api-access-7mss5\") pod \"a692a891-8165-486f-b355-6a53917b00e2\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") "
Sep 18 16:50:13.575264 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.574993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-cabundle-cert\") pod \"a692a891-8165-486f-b355-6a53917b00e2\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") "
Sep 18 16:50:13.575264 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.575030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-kube-rbac-proxy-sar-config\") pod \"a692a891-8165-486f-b355-6a53917b00e2\" (UID: \"a692a891-8165-486f-b355-6a53917b00e2\") "
Sep 18 16:50:13.575369 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.575340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a692a891-8165-486f-b355-6a53917b00e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a692a891-8165-486f-b355-6a53917b00e2" (UID: "a692a891-8165-486f-b355-6a53917b00e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:50:13.575437 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.575378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a692a891-8165-486f-b355-6a53917b00e2" (UID: "a692a891-8165-486f-b355-6a53917b00e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:50:13.575437 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.575405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a692a891-8165-486f-b355-6a53917b00e2" (UID: "a692a891-8165-486f-b355-6a53917b00e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:50:13.577140 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.577120    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a692a891-8165-486f-b355-6a53917b00e2" (UID: "a692a891-8165-486f-b355-6a53917b00e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:50:13.577265 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.577251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a692a891-8165-486f-b355-6a53917b00e2-kube-api-access-7mss5" (OuterVolumeSpecName: "kube-api-access-7mss5") pod "a692a891-8165-486f-b355-6a53917b00e2" (UID: "a692a891-8165-486f-b355-6a53917b00e2"). InnerVolumeSpecName "kube-api-access-7mss5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:50:13.676068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.676025    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a692a891-8165-486f-b355-6a53917b00e2-proxy-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:50:13.676068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.676062    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a692a891-8165-486f-b355-6a53917b00e2-kserve-provision-location\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:50:13.676068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.676073    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mss5\" (UniqueName: \"kubernetes.io/projected/a692a891-8165-486f-b355-6a53917b00e2-kube-api-access-7mss5\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:50:13.676068 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.676083    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-cabundle-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:50:13.676319 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:13.676092    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a692a891-8165-486f-b355-6a53917b00e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\""
Sep 18 16:50:14.389778 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.389745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p" event={"ID":"a692a891-8165-486f-b355-6a53917b00e2","Type":"ContainerDied","Data":"7ef633ca5438ec64722223bbd5776e8c61ab62b800bd91a9f03a3be938da0edc"}
Sep 18 16:50:14.390151 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.389784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"
Sep 18 16:50:14.390151 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.389827    2563 scope.go:117] "RemoveContainer" containerID="897bd5eefea2267b14c84a4279eb2286eb23e66dbf489ffd470ed3bd2d8c80b5"
Sep 18 16:50:14.398303 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.398278    2563 scope.go:117] "RemoveContainer" containerID="7f00880978f3b690196617b77a8eb4429b8ca8c5c3abab001961ce7efeb3fbbe"
Sep 18 16:50:14.405512 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.405493    2563 scope.go:117] "RemoveContainer" containerID="2b2d7386e8b105f80319d5bfba525af8ef8346d725014a5cb0365ac60cd186b0"
Sep 18 16:50:14.410414 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.410391    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"]
Sep 18 16:50:14.412347 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.412324    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-kqp6p"]
Sep 18 16:50:14.414597 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:14.414581    2563 scope.go:117] "RemoveContainer" containerID="2e186ffbba41be29eb14d1d957820dd41932f9d5139659cd2c520eeb7c7bac21"
Sep 18 16:50:16.000661 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:16.000624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:16.164582 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:16.164535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:16.217052 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:16.217019    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a692a891-8165-486f-b355-6a53917b00e2" path="/var/lib/kubelet/pods/a692a891-8165-486f-b355-6a53917b00e2/volumes"
Sep 18 16:50:16.225555 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:16.225536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:18.891104 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:18.891074    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5ws42_dd636e67-a256-4028-b55d-7dbf12f724f7/global-pull-secret-syncer/0.log"
Sep 18 16:50:19.020376 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:19.020346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log"
Sep 18 16:50:19.122126 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:19.122094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-96.ec2.internal_d6e6cf529647eabbf917a54df21705ba/haproxy/0.log"
Sep 18 16:50:22.021695 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:22.021659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:22.076198 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:22.076156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wkrr6_798956e5-559d-4944-b555-f186abf61335/konnectivity-agent/0.log" limit=1024
Sep 18 16:50:23.099019 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.098995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/alertmanager/0.log"
Sep 18 16:50:23.128456 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.128427    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/config-reloader/0.log"
Sep 18 16:50:23.152844 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.152824    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/kube-rbac-proxy-web/0.log"
Sep 18 16:50:23.176482 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.176462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/kube-rbac-proxy/0.log"
Sep 18 16:50:23.202348 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.202287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/kube-rbac-proxy-metric/0.log"
Sep 18 16:50:23.224878 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.224858    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/prom-label-proxy/0.log"
Sep 18 16:50:23.248826 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.248807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4669a34e-9a6a-4471-aced-6747c0f1cd77/init-config-reloader/0.log"
Sep 18 16:50:23.286256 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.286235    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-bllvq_5564b299-dac8-44fa-ac85-4370d08c2bc8/cluster-monitoring-operator/0.log"
Sep 18 16:50:23.398612 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.398583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d586cdf54-5f8fn_50ddce52-3943-4e3d-ac23-24bf7f669fd8/metrics-server/0.log"
Sep 18 16:50:23.429615 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.429590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-zv6kw_b9170ef7-39a5-4934-8499-b2fc1fe927d0/monitoring-plugin/0.log"
Sep 18 16:50:23.471490 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.471420    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7plgk_1a102568-21a5-497b-8a2e-cb67a7f3106f/node-exporter/0.log"
Sep 18 16:50:23.501126 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.501099    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7plgk_1a102568-21a5-497b-8a2e-cb67a7f3106f/kube-rbac-proxy/0.log"
Sep 18 16:50:23.517278 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:23.517251    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7plgk_1a102568-21a5-497b-8a2e-cb67a7f3106f/init-textfile/0.log"
Sep 18 16:50:24.074797 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:24.074765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76f4bd6868-ncgd2_7e17ec68-5d31-4477-a8c8-72ee26b3d0b2/telemeter-client/0.log"
Sep 18 16:50:24.100324 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:24.100297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76f4bd6868-ncgd2_7e17ec68-5d31-4477-a8c8-72ee26b3d0b2/reload/0.log"
Sep 18 16:50:24.120175 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:24.120155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76f4bd6868-ncgd2_7e17ec68-5d31-4477-a8c8-72ee26b3d0b2/kube-rbac-proxy/0.log"
Sep 18 16:50:27.034638 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034603    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"]
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034911    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kube-rbac-proxy"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034922    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kube-rbac-proxy"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034938    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034952    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034959    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034963    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034974    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kserve-container"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034981    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kserve-container"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034987    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="storage-initializer"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.034993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="storage-initializer"
Sep 18 16:50:27.035033 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.035039    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="agent"
Sep 18 16:50:27.035432 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.035047    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kserve-container"
Sep 18 16:50:27.035432 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.035052    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kube-rbac-proxy"
Sep 18 16:50:27.035432 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.035059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a692a891-8165-486f-b355-6a53917b00e2" containerName="kube-rbac-proxy"
Sep 18 16:50:27.035432 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.035066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="132b9401-18e2-4ed8-afef-702ab650d6ba" containerName="kserve-container"
Sep 18 16:50:27.037725 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.037708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.040116 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.040096    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g45m6\"/\"kube-root-ca.crt\""
Sep 18 16:50:27.040253 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.040164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g45m6\"/\"default-dockercfg-wgwnv\""
Sep 18 16:50:27.040717 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.040698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g45m6\"/\"openshift-service-ca.crt\""
Sep 18 16:50:27.048295 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.048274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"]
Sep 18 16:50:27.078915 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.078893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-sys\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.079034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.078922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqwf\" (UniqueName: \"kubernetes.io/projected/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-kube-api-access-bgqwf\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.079034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.078948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-podres\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.079034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.078974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-lib-modules\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.079034 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.079007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-proc\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179322 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-proc\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-sys\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqwf\" (UniqueName: \"kubernetes.io/projected/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-kube-api-access-bgqwf\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-podres\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-sys\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-lib-modules\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179503 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-proc\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179798 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-podres\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179798 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-lib-modules\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.179906 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.179890    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-ldscf_66a92c4b-715b-419d-bcc7-49933ba01916/volume-data-source-validator/0.log"
Sep 18 16:50:27.187451 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.187430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqwf\" (UniqueName: \"kubernetes.io/projected/bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e-kube-api-access-bgqwf\") pod \"perf-node-gather-daemonset-zlkks\" (UID: \"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e\") " pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.347885 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.347799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:27.623408 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:27.623327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"]
Sep 18 16:50:27.627058 ip-10-0-136-96 kubenswrapper[2563]: W0918 16:50:27.627017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd5c9a1d_7aa9_48f5_9ca3_9c49d9e1920e.slice/crio-fc41a9de5bc0a9758c95de7546d862080fa04f39184cc58ef10f02ed54523bbb WatchSource:0}: Error finding container fc41a9de5bc0a9758c95de7546d862080fa04f39184cc58ef10f02ed54523bbb: Status 404 returned error can't find the container with id fc41a9de5bc0a9758c95de7546d862080fa04f39184cc58ef10f02ed54523bbb
Sep 18 16:50:28.208897 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.208860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tss2n_95792173-3a85-4a4e-bcb4-624522af1e2b/dns/0.log"
Sep 18 16:50:28.229413 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.229384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tss2n_95792173-3a85-4a4e-bcb4-624522af1e2b/kube-rbac-proxy/0.log"
Sep 18 16:50:28.275000 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.274962    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmmfb_29305122-d419-448f-9807-b92cc8861913/dns-node-resolver/0.log"
Sep 18 16:50:28.457940 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.457899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks" event={"ID":"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e","Type":"ContainerStarted","Data":"7cffe70c8a9fbde285e34a176fdf418432aa22567df61d262af996ec300b4ac3"}
Sep 18 16:50:28.458148 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.458109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:28.458214 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.458155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks" event={"ID":"bd5c9a1d-7aa9-48f5-9ca3-9c49d9e1920e","Type":"ContainerStarted","Data":"fc41a9de5bc0a9758c95de7546d862080fa04f39184cc58ef10f02ed54523bbb"}
Sep 18 16:50:28.473799 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.473706    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks" podStartSLOduration=1.4736916469999999 podStartE2EDuration="1.473691647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:28.472373902 +0000 UTC m=+2040.472634704" watchObservedRunningTime="2026-09-18 16:50:28.473691647 +0000 UTC m=+2040.473952458"
Sep 18 16:50:28.781374 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:28.781293    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxdtm_312dfd92-b94e-484d-821a-f9ffba57a32f/node-ca/0.log"
Sep 18 16:50:29.552621 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:29.552590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d877f4498-kz5rl_920459df-d1e0-476a-8a98-df1514196669/router/0.log"
Sep 18 16:50:29.925718 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:29.925689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9p64m_5831d0a0-336e-41ba-9365-c73a416d6dd3/serve-healthcheck-canary/0.log"
Sep 18 16:50:30.429492 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:30.429460    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5b8q_454a38f9-b610-48bf-8429-890bc8b5101c/kube-rbac-proxy/0.log"
Sep 18 16:50:30.451831 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:30.451798    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5b8q_454a38f9-b610-48bf-8429-890bc8b5101c/exporter/0.log"
Sep 18 16:50:30.475215 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:30.475188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f5b8q_454a38f9-b610-48bf-8429-890bc8b5101c/extractor/0.log"
Sep 18 16:50:33.064981 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:33.064947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-6xf7d_8c02100f-6f28-49ed-97ce-023a29a93949/manager/0.log"
Sep 18 16:50:34.473165 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:34.473139    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g45m6/perf-node-gather-daemonset-zlkks"
Sep 18 16:50:37.343915 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:37.343831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4pp6f_b0a56faa-83ce-4f22-a03d-8b50ac74f33d/migrator/0.log"
Sep 18 16:50:37.365340 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:37.365314    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-4pp6f_b0a56faa-83ce-4f22-a03d-8b50ac74f33d/graceful-termination/0.log"
Sep 18 16:50:38.961822 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:38.961752    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:50:38.983350 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:38.983321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/egress-router-binary-copy/0.log"
Sep 18 16:50:39.003449 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.003422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/cni-plugins/0.log"
Sep 18 16:50:39.024426 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.024397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/bond-cni-plugin/0.log"
Sep 18 16:50:39.045092 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.045065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/routeoverride-cni/0.log"
Sep 18 16:50:39.067155 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.067130    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/whereabouts-cni-bincopy/0.log"
Sep 18 16:50:39.089188 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.089155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ndq2c_8746475b-5bd4-4eb0-b54a-93ebca3ba19f/whereabouts-cni/0.log"
Sep 18 16:50:39.125866 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.125836    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h2nqg_e5841404-0c91-44f3-9153-111da58f2daa/kube-multus/0.log"
Sep 18 16:50:39.212144 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.212071    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7t76l_66a797a2-9e7c-4c77-9523-2e62b0cccb79/network-metrics-daemon/0.log"
Sep 18 16:50:39.231961 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:39.231937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7t76l_66a797a2-9e7c-4c77-9523-2e62b0cccb79/kube-rbac-proxy/0.log"
Sep 18 16:50:40.040326 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.040292    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-controller/0.log"
Sep 18 16:50:40.058310 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.058279    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/0.log"
Sep 18 16:50:40.058577 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.058548    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovn-acl-logging/1.log"
Sep 18 16:50:40.078793 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.078772    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/kube-rbac-proxy-node/0.log"
Sep 18 16:50:40.102604 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.102578    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 16:50:40.121715 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.121692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/northd/0.log"
Sep 18 16:50:40.142630 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.142605    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/nbdb/0.log"
Sep 18 16:50:40.168227 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.168205    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/sbdb/0.log"
Sep 18 16:50:40.205401 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:40.205379    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fw7m5_64a4bc63-9c28-4830-81d2-8d2dd866b848/ovnkube-controller/0.log"
Sep 18 16:50:41.705259 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:41.705220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bq84s_5bd67021-297e-41c1-b48a-ed9809e9cd2e/network-check-target-container/0.log"
Sep 18 16:50:42.710886 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:42.710855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-s845k_132532f8-4679-4c8c-b617-1b2ac984951a/iptables-alerter/0.log"
Sep 18 16:50:43.398316 ip-10-0-136-96 kubenswrapper[2563]: I0918 16:50:43.398287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rbvbv_981024a2-6f8a-47d8-b8fd-76833fed016b/tuned/0.log"