Sep 17 17:33:05.785675 ip-10-0-135-202 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:33:05.785688 ip-10-0-135-202 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:33:05.785696 ip-10-0-135-202 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:33:05.785904 ip-10-0-135-202 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:33:15.879667 ip-10-0-135-202 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:33:15.879687 ip-10-0-135-202 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f8f5c6b42225414db099d322fceb275f --
Sep 17 17:35:14.370097 ip-10-0-135-202 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:35:14.709082 ip-10-0-135-202 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:35:14.709082 ip-10-0-135-202 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:35:14.709082 ip-10-0-135-202 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:35:14.709082 ip-10-0-135-202 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:35:14.709082 ip-10-0-135-202 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:35:14.711063 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.710981    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:35:14.714687 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714671    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:14.714687 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714686    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714691    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714694    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714698    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714701    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714704    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714707    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714710    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714714    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714716    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714720    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714725    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714727    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714732    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714736    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714739    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714742    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714744    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714747    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:14.714754 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714755    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714758    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714761    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714764    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714767    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714769    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714773    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714775    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714778    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714780    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714783    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714785    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714799    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714802    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714806    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714809    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714811    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714814    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714816    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:14.715226 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714819    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714822    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714824    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714827    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714829    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714832    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714835    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714838    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714840    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714843    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714845    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714848    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714850    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714853    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714859    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714863    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714865    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714868    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714872    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714876    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:14.715678 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714879    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714882    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714885    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714887    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714890    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714893    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714895    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714898    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714900    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714903    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714905    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714908    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714911    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714914    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714917    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714919    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714922    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714925    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714927    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714930    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:14.716175 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714933    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714935    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714938    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714941    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714944    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714947    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714949    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714953    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714956    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714959    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714961    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714964    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714967    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714969    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714972    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714975    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714977    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714980    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714983    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.714986    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:14.716653 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715657    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715663    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715666    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715668    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715671    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715674    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715677    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715680    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715683    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715686    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715689    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715692    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715695    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715697    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715700    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715703    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715705    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715708    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715711    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715714    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:14.717155 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715717    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715719    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715722    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715725    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715727    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715730    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715733    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715735    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715738    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715740    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715743    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715745    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715748    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715752    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715756    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715759    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715762    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715764    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715767    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715770    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:14.717651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715773    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715775    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715778    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715781    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715783    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715785    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715804    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715807    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715810    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715813    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715817    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715819    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715822    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715825    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715828    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715831    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715833    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715836    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715839    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715842    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:14.718161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715845    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715848    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715850    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715852    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715855    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715857    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715860    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715862    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715865    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715867    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715870    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715874    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715877    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715881    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715884    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715887    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715890    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715892    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715895    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:14.718654 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715899    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715902    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715905    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715907    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715910    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715912    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715915    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715918    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715921    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715923    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715926    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715928    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715931    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715933    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715936    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715940    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715944    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715947    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715950    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:14.719126 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.715953    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716298    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716303    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716306    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716309    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716311    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716314    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716317    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716319    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716322    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716325    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716327    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716330    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716335    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716338    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716341    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716344    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716346    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716349    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716351    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:14.719658 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716354    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716357    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716360    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716362    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716365    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716368    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716370    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716373    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716375    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716378    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716380    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716383    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716385    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716388    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716391    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716393    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716396    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716398    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716401    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716403    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:14.720171 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716406    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716409    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716412    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716416    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716420    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716424    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716427    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716430    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716433    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716436    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716438    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716441    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716444    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716447    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716450    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716453    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716456    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716458    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716461    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:14.720651 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716464    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716466    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716469    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716471    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716474    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716479    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716482    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716484    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716487    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716489    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716492    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716494    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716497    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716499    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716502    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716504    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716507    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716509    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716514    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:14.721131 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716517    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716520    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716523    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716526    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716529    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716532    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716534    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716538    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716541    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716543    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716546    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716549    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716551    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716554    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716556    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716559    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716562    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716564    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716568    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:14.721646 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716570    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716573    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.716575    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716648    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716678    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716685    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716689    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716694    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716697    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716702    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716706    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716709    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716714    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716717    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716721    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716724    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716727    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716730    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716733    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716736    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716739    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716743    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716747    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716750    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:35:14.722137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716753    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716756    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716765    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716768    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716772    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716775    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716778    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716783    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716786    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716801    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716804    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716809    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716812    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716815    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716818    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716821    2563 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716824    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716828    2563 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716831    2563 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716834    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716839    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716851    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716856    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716859    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716862    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:35:14.722726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716865    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716868    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716871    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716874    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716876    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716879    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716883    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716886    2563 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716889    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716892    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716895    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716898    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716901    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716904    2563 flags.go:64] FLAG: --help="false"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716908    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-135-202.ec2.internal"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716911    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716914    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716917    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716921    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716924    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716927    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716930    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716932    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:35:14.723405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716935    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716938    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716941    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716944    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716949    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716951    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716955    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716957    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716960    2563 flags.go:64] FLAG: --lock-file=""
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716963    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716966    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716969    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716974    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716977    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716980    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716983    2563 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716986    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716990    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716992    2563 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716995    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.716999    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717002    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717006    2563 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717011    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717014    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:35:14.723990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717016    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717019    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717022    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717026    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717029    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717036    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717039    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717042    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717045    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717048    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717053    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717056    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717061    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717063    2563 flags.go:64] FLAG: --port="10250"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717066    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717069    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03070d0f102dedc14"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717072    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717075    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717078    2563 flags.go:64] FLAG: --register-node="true"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717081    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717084    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717088    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717090    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717093    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717097    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:35:14.724666 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717100    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717103    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717106    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717109    2563 flags.go:64] FLAG: --runonce="false"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717112    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717116    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717119    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717122    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717125    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717128    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717131    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717134    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717137    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717140    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717143    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717145    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717148    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717151    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717154    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717160    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717163    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717166    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717170    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717174    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717176    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:35:14.725632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717179    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717182    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717185    2563 flags.go:64] FLAG: --v="2"
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717189    2563 flags.go:64] FLAG: --version="false"
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717193    2563 flags.go:64] FLAG: --vmodule=""
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717196    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.717199    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717317    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717321    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717325    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717328    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717331    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717335    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717338    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717340    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717343    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717345    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717348    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717351    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717353    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717356    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:14.726745 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717358    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717361    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717363    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717366    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717369    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717374    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717378    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717381    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717385    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717390    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717393    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717396    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717399    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717401    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717405    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717408    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717410    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717413    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717415    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717419    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:14.727785 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717422    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717424    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717427    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717431    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717434    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717436    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717439    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717442    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717445    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717447    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717450    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717454    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717458    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717461    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717464    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717467    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717470    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717474    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717477    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:14.728409 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717480    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717482    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717485    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717487    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717490    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717493    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717495    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717498    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717501    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717504    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717506    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717509    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717511    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717514    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717518    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717521    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717526    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717530    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717533    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:14.728928 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717536    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717539    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717541    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717544    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717547    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717549    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717552    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717554    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717557    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717560    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717562    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717567    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717569    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717572    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717575    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717578    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717580    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717583    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717585    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717588    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:14.729395 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717591    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717593    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717596    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717598    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717601    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717604    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.717606    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.718205    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.728479    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.728503    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.728524    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.728533    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728593    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728599    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728602    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:14.729905 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728605    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728608    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728611    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728614    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728618    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728621    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728623    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728626    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728629    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728632    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728634    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728637    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728640    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728642    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728645    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728648    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728651    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728653    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728656    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:14.730286 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728659    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728661    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728664    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728666    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728670    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728672    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728676    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728679    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728681    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728684    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728687    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728690    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728693    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728696    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728698    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728701    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728704    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728707    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728710    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728712    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:14.730747 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728715    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728718    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728720    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728723    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728725    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728728    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728731    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728733    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728736    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728739    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728741    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728744    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728746    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728749    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728751    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728754    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728757    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728759    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728762    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728766    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:14.731321 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728768    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728771    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728774    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728776    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728779    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728781    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728784    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728786    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728806    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728809    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728812    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728815    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728818    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728822    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728826    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728829    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728832    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728838    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:14.731865 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728848    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728852    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728855    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728858    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728861    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728864    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728868    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728871    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728874    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728876    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728879    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728882    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728885    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728888    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728891    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728893    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728896    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728899    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:14.732304 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.728902    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.728907    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729025    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729030    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729033    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729036    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729039    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729043    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729047    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729050    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729053    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729056    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729059    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729061    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:14.732783 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729065    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729068    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729071    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729074    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729077    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729080    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729083    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729085    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729088    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729091    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729093    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729096    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729099    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729102    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729105    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729108    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729111    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729113    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729116    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729119    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:14.733161 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729122    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729124    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729127    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729130    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729132    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729135    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729137    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729140    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729143    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729145    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729149    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729153    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729156    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729159    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729162    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729164    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729167    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729170    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729173    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:14.733676 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729175    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729179    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729182    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729185    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729188    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729191    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729194    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729197    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729199    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729202    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729204    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729207    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729210    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729212    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729215    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729217    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729220    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729222    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729225    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:14.734150 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729227    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729230    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729233    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729235    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729238    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729240    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729243    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729245    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729248    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729251    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729253    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729256    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729259    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729261    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729264    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729266    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729269    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729271    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729275    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729278    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:14.734613 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729280    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729283    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729286    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729288    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729291    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729294    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729296    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729299    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:14.729302    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.729307    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.729428    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.732486    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.733178    2563 server.go:1019] "Starting client certificate rotation"
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.733275    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:35:14.735160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.733318    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:35:14.736255 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.736239    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:35:14.738174 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.738153    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:35:14.748441 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.748420    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:35:14.751145 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.751125    2563 log.go:25] "Validated CRI v1 image API"
Sep 17 17:35:14.751990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.751968    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:35:14.755532 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.755508    2563 fs.go:135] Filesystem UUIDs: map[42b09659-bcea-4f59-b9ff-669d7e5a12c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bc521eb0-c747-44cc-9f2c-735c37895fde:/dev/nvme0n1p4]
Sep 17 17:35:14.755630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.755531    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:35:14.761903 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.761760    2563 manager.go:216] Machine: {Timestamp:2026-09-17 17:35:14.759750288 +0000 UTC m=+0.287532105 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3081879 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce4996300df98899fd3470afe08d1 SystemUUID:ec2ce499-6300-df98-899f-d3470afe08d1 BootID:f8f5c6b4-2225-414d-b099-d322fceb275f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:a8:19:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:a8:19:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:7c:06:bf:b5:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:35:14.761903 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.761895    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:35:14.762072 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.762009    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:35:14.764917 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.764887    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:35:14.765409 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.764923    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:35:14.766019 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.766007    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:35:14.766076 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.766022    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:35:14.766076 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.766039    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:35:14.766150 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.766133    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:35:14.766663 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.766651    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:35:14.768196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.768185    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:35:14.771761 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.771734    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:35:14.772076 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772064    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:35:14.772109 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772104    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:35:14.772150 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772117    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:35:14.772150 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772132    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:35:14.772150 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772144    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:35:14.772251 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772156    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:35:14.772602 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772589    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:35:14.772649 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.772613    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:35:14.774472 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.774457    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:35:14.776971 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.776957    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:35:14.777026 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.776988    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:35:14.777903 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777892    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:35:14.777945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777908    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:35:14.777945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777915    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:35:14.777945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777920    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:35:14.777945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777929    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:35:14.777945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777935    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:35:14.778069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777949    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:35:14.778069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777954    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:35:14.778069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777961    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:35:14.778069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777967    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:35:14.778069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777974    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:35:14.778069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.777980    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:35:14.779253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.779242    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:35:14.779253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.779253    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:35:14.782972 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.782956    2563 server.go:1268] "Started kubelet"
Sep 17 17:35:14.783077 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.783044    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:35:14.783121 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.783063    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:35:14.783159 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.783137    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:35:14.783915 ip-10-0-135-202 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:35:14.784811 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.784599    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:35:14.785093 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.785080    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:35:14.790092 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.790065    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:35:14.790888 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.790873    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:35:14.791021 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.790987    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:35:14.791095 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.791080    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:35:14.791587 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.790749    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-202.ec2.internal.18d62c36900c45fe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-202.ec2.internal,UID:ip-10-0-135-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-202.ec2.internal,},FirstTimestamp:2026-09-17 17:35:14.78293043 +0000 UTC m=+0.310712247,LastTimestamp:2026-09-17 17:35:14.78293043 +0000 UTC m=+0.310712247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-202.ec2.internal,}"
Sep 17 17:35:14.793935 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.793916    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:35:14.794297 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.794283    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:35:14.794489 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.794471    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:14.794921 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.794902    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:35:14.795020 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.794924    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:35:14.795020 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.794940    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:35:14.795114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795053    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:35:14.795114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795063    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:35:14.795114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795094    2563 factory.go:55] Registering systemd factory
Sep 17 17:35:14.795235 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795133    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:35:14.795371 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795358    2563 factory.go:153] Registering CRI-O factory
Sep 17 17:35:14.795407 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795376    2563 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:35:14.795440 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795425    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:35:14.795477 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795443    2563 factory.go:103] Registering Raw factory
Sep 17 17:35:14.795477 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795459    2563 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:35:14.795881 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.795868    2563 manager.go:349] Starting recovery of all containers
Sep 17 17:35:14.797712 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.797685    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:35:14.797835 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.797736    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:35:14.808729 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.808484    2563 manager.go:354] Recovery completed
Sep 17 17:35:14.809918 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.809880    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 17:35:14.811711 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.811679    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kgvn5"
Sep 17 17:35:14.813855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.813840    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:14.817771 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.817754    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:14.817857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.817787    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:14.817857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.817820    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:14.818495 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.818478    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:14.818578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.818478    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:35:14.818578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.818533    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:35:14.818578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.818566    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:35:14.819509 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.819491    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:14.819604 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.819524    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:14.819604 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.819538    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:14.819816 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.819766    2563 policy_none.go:49] "None policy: Start"
Sep 17 17:35:14.819816 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.819784    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:35:14.819816 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.819814    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:35:14.820265 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.820169    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-202.ec2.internal.18d62c36921fe582  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-202.ec2.internal,UID:ip-10-0-135-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-202.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-202.ec2.internal,},FirstTimestamp:2026-09-17 17:35:14.817770882 +0000 UTC m=+0.345552699,LastTimestamp:2026-09-17 17:35:14.817770882 +0000 UTC m=+0.345552699,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-202.ec2.internal,}"
Sep 17 17:35:14.820507 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.820487    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kgvn5"
Sep 17 17:35:14.822240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.822227    2563 policy_none.go:47] "Start"
Sep 17 17:35:14.850519 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.850502    2563 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.850547    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.850575    2563 server.go:87] "Starting device plugin registration server"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.850932    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.850952    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.851040    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.851136    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.851147    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.851873    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:35:14.863405 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.851924    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:14.950578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.950535    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:35:14.951028 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.951012    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:14.951857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.951839    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:35:14.951916 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.951903    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:35:14.951962 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.951937    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:35:14.952007 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.951989    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:35:14.953925 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.953904    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:14.954075 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.953938    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:14.954075 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.953953    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:14.954075 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.953982    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-202.ec2.internal"
Sep 17 17:35:14.954439 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.954421    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:14.960927 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:14.960854    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-202.ec2.internal"
Sep 17 17:35:14.960927 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.960882    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-202.ec2.internal\": node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:14.973501 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:14.973474    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.052914 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.052879    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"]
Sep 17 17:35:15.053049 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.052978    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:15.054379 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.054365    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:15.054448 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.054393    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:15.054448 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.054403    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:15.055757 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.055745    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:15.056257 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.056243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.056303 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.056272    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:15.056570 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.056556    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:15.056629 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.056581    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:15.056629 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.056592    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:15.057247 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.057233    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:15.057350 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.057260    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:15.057350 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.057273    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:15.057661 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.057641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.057661 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.057666    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:15.058652 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.058631    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:15.058715 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.058656    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:15.058715 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.058665    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:15.073966 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.073945    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.080386 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.080371    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-202.ec2.internal\" not found" node="ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.084986 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.084970    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-202.ec2.internal\" not found" node="ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.096180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.096148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8870edaf0c4a8354936a672f9fccc76-config\") pod \"kube-apiserver-proxy-ip-10-0-135-202.ec2.internal\" (UID: \"a8870edaf0c4a8354936a672f9fccc76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.096180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.096182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a060ffcaba569572d8e2fac80d1651df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal\" (UID: \"a060ffcaba569572d8e2fac80d1651df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.096315 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.096209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a060ffcaba569572d8e2fac80d1651df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal\" (UID: \"a060ffcaba569572d8e2fac80d1651df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.174057 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.174028    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.197279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.197254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8870edaf0c4a8354936a672f9fccc76-config\") pod \"kube-apiserver-proxy-ip-10-0-135-202.ec2.internal\" (UID: \"a8870edaf0c4a8354936a672f9fccc76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.197279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.197279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a060ffcaba569572d8e2fac80d1651df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal\" (UID: \"a060ffcaba569572d8e2fac80d1651df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.197435 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.197303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a060ffcaba569572d8e2fac80d1651df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal\" (UID: \"a060ffcaba569572d8e2fac80d1651df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.197435 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.197333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a060ffcaba569572d8e2fac80d1651df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal\" (UID: \"a060ffcaba569572d8e2fac80d1651df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.197435 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.197361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a060ffcaba569572d8e2fac80d1651df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal\" (UID: \"a060ffcaba569572d8e2fac80d1651df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.197435 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.197360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8870edaf0c4a8354936a672f9fccc76-config\") pod \"kube-apiserver-proxy-ip-10-0-135-202.ec2.internal\" (UID: \"a8870edaf0c4a8354936a672f9fccc76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.274679 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.274615    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.375263 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.375231    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.385110 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.385080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.388887 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.388858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:15.478301 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.478251    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.580343 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.580272    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.686975 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.686943    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.733806 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.733757    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:35:15.734285 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.733976    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:35:15.788163 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.788123    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.826209 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.826174    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:30:15 +0000 UTC" deadline="2028-03-13 14:31:07.589811708 +0000 UTC"
Sep 17 17:35:15.826209 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.826206    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13028h55m51.76361016s"
Sep 17 17:35:15.889247 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.889211    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:15.908784 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.908753    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:15.973026 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:15.972991    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda060ffcaba569572d8e2fac80d1651df.slice/crio-2c9c88fdb85a89b1b5409d6ea06390d4125f9632882f2e0bae7197cf8e74bc03 WatchSource:0}: Error finding container 2c9c88fdb85a89b1b5409d6ea06390d4125f9632882f2e0bae7197cf8e74bc03: Status 404 returned error can't find the container with id 2c9c88fdb85a89b1b5409d6ea06390d4125f9632882f2e0bae7197cf8e74bc03
Sep 17 17:35:15.973593 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:15.973570    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8870edaf0c4a8354936a672f9fccc76.slice/crio-a78eba58e14e50789bcdf35ce16449409e0f43929490acee145394505119c57c WatchSource:0}: Error finding container a78eba58e14e50789bcdf35ce16449409e0f43929490acee145394505119c57c: Status 404 returned error can't find the container with id a78eba58e14e50789bcdf35ce16449409e0f43929490acee145394505119c57c
Sep 17 17:35:15.978804 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:15.978787    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:35:15.989430 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:15.989402    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:16.095961 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.095859    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:16.141642 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.141614    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:16.196721 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.196688    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-202.ec2.internal\" not found"
Sep 17 17:35:16.217464 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.217440    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:16.295899 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.295863    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:16.310012 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.309849    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:35:16.311177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.311162    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal"
Sep 17 17:35:16.322992 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.322975    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:35:16.658169 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.658134    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:16.773396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.773362    2563 apiserver.go:52] "Watching apiserver"
Sep 17 17:35:16.781542 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.781501    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:35:16.782086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.782059    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7swdh","kube-system/global-pull-secret-syncer-sd6ff","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal","openshift-multus/network-metrics-daemon-zxjft","openshift-network-operator/iptables-alerter-bxb95","openshift-ovn-kubernetes/ovnkube-node-jmbb8","kube-system/konnectivity-agent-btktp","kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tnnpz","openshift-dns/node-resolver-lwf8b","openshift-image-registry/node-ca-t5jhz","openshift-multus/multus-additional-cni-plugins-knj9c","openshift-multus/multus-chk69"]
Sep 17 17:35:16.783810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.783745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.785209 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.785188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.787902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.787791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.789484 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.789063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.789937 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.789656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.790567 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.790567 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.790567 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.790567 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lxj26\""
Sep 17 17:35:16.790567 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:35:16.790567 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790522    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ngq6c\""
Sep 17 17:35:16.790939 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.790740    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:35:16.791109 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.791089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:16.791172 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.791161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.791221 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.791161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:16.791630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.791310    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:35:16.791630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.791493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s8nlb\""
Sep 17 17:35:16.791630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.791508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:35:16.791630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.791578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.792830 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.792798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.796733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.794202    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:35:16.796733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.794603    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ml89q\""
Sep 17 17:35:16.796733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.794768    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:35:16.796733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.795615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.797428 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.797220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.798340 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.798321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:35:16.798991 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.798968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:16.799437 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.799372    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:16.800109 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.800088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f9fsm\""
Sep 17 17:35:16.800924 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.800891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:16.801018 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.800926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.801018 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.800957    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:16.801691 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.801671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:35:16.802486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.802466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:16.803742 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.803708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:35:16.804162 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.804134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:16.804491 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.804474    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ncrnk\""
Sep 17 17:35:16.804578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.804515    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.804798 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.804779    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.804953 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.804934    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.805491 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g5qvn\""
Sep 17 17:35:16.805491 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:35:16.805491 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:35:16.805491 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:35:16.805755 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805590    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3921a5cf-6472-434c-a537-f1658c67a4d1-iptables-alerter-script\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3921a5cf-6472-434c-a537-f1658c67a4d1-host-slash\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0df6d708-1540-4e39-b75d-cfcc8395a5f8-tmp\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-cni-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.805962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-k8s-cni-cncf-io\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-cni-multus\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cnibin\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-lib-modules\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggqz\" (UniqueName: \"kubernetes.io/projected/0df6d708-1540-4e39-b75d-cfcc8395a5f8-kube-api-access-tggqz\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-socket-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.806234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-cni-bin\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-multus-certs\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94llw\" (UniqueName: \"kubernetes.io/projected/4d3b541d-198a-49f5-be28-85d0ea5f1c88-kube-api-access-94llw\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cni-binary-copy\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj65\" (UniqueName: \"kubernetes.io/projected/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-kube-api-access-tqj65\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-netns\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-hostroot\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-var-lib-kubelet\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aacb1a6-3867-4f18-8241-4d9e294d8db0-konnectivity-ca\") pod \"konnectivity-agent-btktp\" (UID: \"1aacb1a6-3867-4f18-8241-4d9e294d8db0\") " pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-modprobe-d\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-tuned\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-registration-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-sys-fs\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysctl-conf\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-systemd\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.806957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pdg\" (UniqueName: \"kubernetes.io/projected/8538cafd-8176-42c0-bbe3-4370a998174e-kube-api-access-k4pdg\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-cnibin\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-kubelet\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-conf-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-system-cni-dir\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysctl-d\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-socket-dir-parent\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-daemon-config\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806909    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.806925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-etc-kubernetes\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhhd\" (UniqueName: \"kubernetes.io/projected/d2d139ed-fedb-42f4-b2d7-10e810b16090-kube-api-access-xhhhd\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-kubernetes\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-sys\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-system-cni-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.807694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-host\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.808511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mpc\" (UniqueName: \"kubernetes.io/projected/3921a5cf-6472-434c-a537-f1658c67a4d1-kube-api-access-h7mpc\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.808511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aacb1a6-3867-4f18-8241-4d9e294d8db0-agent-certs\") pod \"konnectivity-agent-btktp\" (UID: \"1aacb1a6-3867-4f18-8241-4d9e294d8db0\") " pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.808511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysconfig\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.808511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-run\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.808511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-device-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.808511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.807382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d3b541d-198a-49f5-be28-85d0ea5f1c88-cni-binary-copy\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.810090 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.810070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:35:16.810175 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.810126    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqqbm\""
Sep 17 17:35:16.810175 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.810148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:35:16.810289 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.810189    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:35:16.810442 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.810426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9q944\""
Sep 17 17:35:16.828001 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.827826    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:35:16.896787 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.896760    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:35:16.907126 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.907094    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttwlm"
Sep 17 17:35:16.908279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-cnibin\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.908279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-conf-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.908279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4338b764-2fc9-42cf-a3a2-61f47006dbbd-serviceca\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-socket-dir-parent\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-sys\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-systemd-units\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908367    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnx7\" (UniqueName: \"kubernetes.io/projected/ecef2930-3fcc-4dbb-b497-f58796174663-kube-api-access-bcnx7\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-host\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-conf-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mpc\" (UniqueName: \"kubernetes.io/projected/3921a5cf-6472-434c-a537-f1658c67a4d1-kube-api-access-h7mpc\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-cnibin\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aacb1a6-3867-4f18-8241-4d9e294d8db0-agent-certs\") pod \"konnectivity-agent-btktp\" (UID: \"1aacb1a6-3867-4f18-8241-4d9e294d8db0\") " pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysconfig\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d3b541d-198a-49f5-be28-85d0ea5f1c88-cni-binary-copy\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-host\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.908504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-run-netns\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-socket-dir-parent\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3921a5cf-6472-434c-a537-f1658c67a4d1-host-slash\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3921a5cf-6472-434c-a537-f1658c67a4d1-host-slash\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-sys\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-k8s-cni-cncf-io\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysconfig\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.909268 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-cni-multus\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.908827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7k7w\" (UniqueName: \"kubernetes.io/projected/6db84866-0cdb-4b78-b53d-5c019281e212-kube-api-access-n7k7w\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-cni-multus\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-k8s-cni-cncf-io\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cnibin\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-lib-modules\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cnibin\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d3b541d-198a-49f5-be28-85d0ea5f1c88-cni-binary-copy\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-ovnkube-config\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db84866-0cdb-4b78-b53d-5c019281e212-hosts-file\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cni-binary-copy\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj65\" (UniqueName: \"kubernetes.io/projected/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-kube-api-access-tqj65\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05a68412-83e9-486b-a686-65730e2448ba-dbus\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-lib-modules\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-etc-selinux\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.909785 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4338b764-2fc9-42cf-a3a2-61f47006dbbd-host\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909764    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-var-lib-kubelet\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-registration-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.909944    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-var-lib-kubelet\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.909990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-ovnkube-script-lib\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:16.910029    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.409988823 +0000 UTC m=+2.937770628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-registration-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysctl-conf\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cni-binary-copy\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05a68412-83e9-486b-a686-65730e2448ba-kubelet-config\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-system-cni-dir\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysctl-conf\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-system-cni-dir\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.910611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysctl-d\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-daemon-config\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-etc-kubernetes\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-var-lib-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-etc-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-sysctl-d\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-etc-kubernetes\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhhd\" (UniqueName: \"kubernetes.io/projected/d2d139ed-fedb-42f4-b2d7-10e810b16090-kube-api-access-xhhhd\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-kubernetes\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-system-cni-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-node-log\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-kubernetes\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.910987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-daemon-config\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.911483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-cni-netd\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-system-cni-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-env-overrides\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-run\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-device-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-systemd\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-device-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-ovn\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-run\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-cni-bin\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvk2\" (UniqueName: \"kubernetes.io/projected/4338b764-2fc9-42cf-a3a2-61f47006dbbd-kube-api-access-stvk2\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3921a5cf-6472-434c-a537-f1658c67a4d1-iptables-alerter-script\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0df6d708-1540-4e39-b75d-cfcc8395a5f8-tmp\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-cni-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.912382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-multus-cni-dir\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db84866-0cdb-4b78-b53d-5c019281e212-tmp-dir\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggqz\" (UniqueName: \"kubernetes.io/projected/0df6d708-1540-4e39-b75d-cfcc8395a5f8-kube-api-access-tggqz\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-socket-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-cni-bin\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-multus-certs\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94llw\" (UniqueName: \"kubernetes.io/projected/4d3b541d-198a-49f5-be28-85d0ea5f1c88-kube-api-access-94llw\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.911992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-netns\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-hostroot\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-kubelet\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-slash\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-log-socket\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-kubelet\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aacb1a6-3867-4f18-8241-4d9e294d8db0-konnectivity-ca\") pod \"konnectivity-agent-btktp\" (UID: \"1aacb1a6-3867-4f18-8241-4d9e294d8db0\") " pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-modprobe-d\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-tuned\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-sys-fs\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.913278 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ecef2930-3fcc-4dbb-b497-f58796174663-ovn-node-metrics-cert\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-systemd\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pdg\" (UniqueName: \"kubernetes.io/projected/8538cafd-8176-42c0-bbe3-4370a998174e-kube-api-access-k4pdg\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-modprobe-d\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-kubelet\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-netns\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3921a5cf-6472-434c-a537-f1658c67a4d1-iptables-alerter-script\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-hostroot\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.912924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-socket-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.913094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-var-lib-cni-bin\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.913133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d3b541d-198a-49f5-be28-85d0ea5f1c88-host-run-multus-certs\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.913280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1aacb1a6-3867-4f18-8241-4d9e294d8db0-konnectivity-ca\") pod \"konnectivity-agent-btktp\" (UID: \"1aacb1a6-3867-4f18-8241-4d9e294d8db0\") " pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.913325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-sys-fs\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.913344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8538cafd-8176-42c0-bbe3-4370a998174e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.914177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.913367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-systemd\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.915923 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.915898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0df6d708-1540-4e39-b75d-cfcc8395a5f8-etc-tuned\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.916273 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.916165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1aacb1a6-3867-4f18-8241-4d9e294d8db0-agent-certs\") pod \"konnectivity-agent-btktp\" (UID: \"1aacb1a6-3867-4f18-8241-4d9e294d8db0\") " pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:16.916369 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.916360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0df6d708-1540-4e39-b75d-cfcc8395a5f8-tmp\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:16.930928 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.930873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mpc\" (UniqueName: \"kubernetes.io/projected/3921a5cf-6472-434c-a537-f1658c67a4d1-kube-api-access-h7mpc\") pod \"iptables-alerter-bxb95\" (UID: \"3921a5cf-6472-434c-a537-f1658c67a4d1\") " pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:16.933557 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.933482    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttwlm"
Sep 17 17:35:16.945010 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.944667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhhd\" (UniqueName: \"kubernetes.io/projected/d2d139ed-fedb-42f4-b2d7-10e810b16090-kube-api-access-xhhhd\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:16.945195 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.945170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94llw\" (UniqueName: \"kubernetes.io/projected/4d3b541d-198a-49f5-be28-85d0ea5f1c88-kube-api-access-94llw\") pod \"multus-chk69\" (UID: \"4d3b541d-198a-49f5-be28-85d0ea5f1c88\") " pod="openshift-multus/multus-chk69"
Sep 17 17:35:16.947143 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.947119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pdg\" (UniqueName: \"kubernetes.io/projected/8538cafd-8176-42c0-bbe3-4370a998174e-kube-api-access-k4pdg\") pod \"aws-ebs-csi-driver-node-9plzn\" (UID: \"8538cafd-8176-42c0-bbe3-4370a998174e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:16.958512 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.958448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal" event={"ID":"a060ffcaba569572d8e2fac80d1651df","Type":"ContainerStarted","Data":"2c9c88fdb85a89b1b5409d6ea06390d4125f9632882f2e0bae7197cf8e74bc03"}
Sep 17 17:35:16.959957 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.959896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal" event={"ID":"a8870edaf0c4a8354936a672f9fccc76","Type":"ContainerStarted","Data":"a78eba58e14e50789bcdf35ce16449409e0f43929490acee145394505119c57c"}
Sep 17 17:35:16.966036 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.966010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj65\" (UniqueName: \"kubernetes.io/projected/2d5c05eb-c75a-45ad-8f2a-06ea33b0005d-kube-api-access-tqj65\") pod \"multus-additional-cni-plugins-knj9c\" (UID: \"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d\") " pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:16.967045 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:16.966987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggqz\" (UniqueName: \"kubernetes.io/projected/0df6d708-1540-4e39-b75d-cfcc8395a5f8-kube-api-access-tggqz\") pod \"tuned-tnnpz\" (UID: \"0df6d708-1540-4e39-b75d-cfcc8395a5f8\") " pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:17.013555 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-ovn\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-cni-bin\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvk2\" (UniqueName: \"kubernetes.io/projected/4338b764-2fc9-42cf-a3a2-61f47006dbbd-kube-api-access-stvk2\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-ovn\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-cni-bin\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db84866-0cdb-4b78-b53d-5c019281e212-tmp-dir\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-kubelet\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.013737    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:17.013760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-slash\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-log-socket\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ecef2930-3fcc-4dbb-b497-f58796174663-ovn-node-metrics-cert\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4338b764-2fc9-42cf-a3a2-61f47006dbbd-serviceca\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.013894    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.513867847 +0000 UTC m=+3.041649665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-systemd-units\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.013984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnx7\" (UniqueName: \"kubernetes.io/projected/ecef2930-3fcc-4dbb-b497-f58796174663-kube-api-access-bcnx7\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-run-netns\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db84866-0cdb-4b78-b53d-5c019281e212-tmp-dir\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-systemd-units\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7k7w\" (UniqueName: \"kubernetes.io/projected/6db84866-0cdb-4b78-b53d-5c019281e212-kube-api-access-n7k7w\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-ovnkube-config\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-slash\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db84866-0cdb-4b78-b53d-5c019281e212-hosts-file\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-log-socket\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.014313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05a68412-83e9-486b-a686-65730e2448ba-dbus\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4338b764-2fc9-42cf-a3a2-61f47006dbbd-host\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-run-netns\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-ovnkube-script-lib\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05a68412-83e9-486b-a686-65730e2448ba-kubelet-config\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4338b764-2fc9-42cf-a3a2-61f47006dbbd-serviceca\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-var-lib-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05a68412-83e9-486b-a686-65730e2448ba-kubelet-config\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-etc-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-etc-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-node-log\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-var-lib-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-cni-netd\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-cni-netd\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-openvswitch\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014553    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-node-log\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05a68412-83e9-486b-a686-65730e2448ba-dbus\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-env-overrides\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db84866-0cdb-4b78-b53d-5c019281e212-hosts-file\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-systemd\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-host-kubelet\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4338b764-2fc9-42cf-a3a2-61f47006dbbd-host\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-ovnkube-config\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ecef2930-3fcc-4dbb-b497-f58796174663-run-systemd\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.014761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-ovnkube-script-lib\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.015855 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.015058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ecef2930-3fcc-4dbb-b497-f58796174663-env-overrides\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.016929 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.016905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ecef2930-3fcc-4dbb-b497-f58796174663-ovn-node-metrics-cert\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.047781 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.047747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvk2\" (UniqueName: \"kubernetes.io/projected/4338b764-2fc9-42cf-a3a2-61f47006dbbd-kube-api-access-stvk2\") pod \"node-ca-t5jhz\" (UID: \"4338b764-2fc9-42cf-a3a2-61f47006dbbd\") " pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.061626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.061597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnx7\" (UniqueName: \"kubernetes.io/projected/ecef2930-3fcc-4dbb-b497-f58796174663-kube-api-access-bcnx7\") pod \"ovnkube-node-jmbb8\" (UID: \"ecef2930-3fcc-4dbb-b497-f58796174663\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.067249 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.067224    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:17.067359 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.067255    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:17.067359 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.067274    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:17.067359 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.067337    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.567320383 +0000 UTC m=+3.095102189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:17.068702 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.068679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7k7w\" (UniqueName: \"kubernetes.io/projected/6db84866-0cdb-4b78-b53d-5c019281e212-kube-api-access-n7k7w\") pod \"node-resolver-lwf8b\" (UID: \"6db84866-0cdb-4b78-b53d-5c019281e212\") " pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.101206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.101178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnnpz"
Sep 17 17:35:17.111100 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.111072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-knj9c"
Sep 17 17:35:17.119794 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.119773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn"
Sep 17 17:35:17.126710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.126687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-chk69"
Sep 17 17:35:17.135594 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.135568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bxb95"
Sep 17 17:35:17.144174 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.144150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:17.151856 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.151833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:17.159562 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.159514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lwf8b"
Sep 17 17:35:17.167237 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.167216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t5jhz"
Sep 17 17:35:17.417571 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.417486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:17.417714 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.417616    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:17.417714 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.417677    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:18.417660872 +0000 UTC m=+3.945442677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:17.518205 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.518166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.518414 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.518334    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:17.518414 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.518411    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:18.518389761 +0000 UTC m=+4.046171592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:17.619506 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.619471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:17.619680 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.619630    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:17.619680 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.619646    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:17.619680 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.619656    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:17.619853 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.619726    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:18.619707092 +0000 UTC m=+4.147488917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:17.936505 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.936443    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:30:17 +0000 UTC" deadline="2028-02-22 11:33:18.159234442 +0000 UTC"
Sep 17 17:35:17.936505 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.936496    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12545h58m0.222741045s"
Sep 17 17:35:17.953629 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.952934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:17.953629 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.953080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:17.953629 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:17.953169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:17.953629 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:17.953253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:17.976499 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:17.976466    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4338b764_2fc9_42cf_a3a2_61f47006dbbd.slice/crio-7198864ebf2db2c8b028a2535d4730ae2083c36e3a9b7b203cd3883f7e0ffc8e WatchSource:0}: Error finding container 7198864ebf2db2c8b028a2535d4730ae2083c36e3a9b7b203cd3883f7e0ffc8e: Status 404 returned error can't find the container with id 7198864ebf2db2c8b028a2535d4730ae2083c36e3a9b7b203cd3883f7e0ffc8e
Sep 17 17:35:17.977668 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:17.977639    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df6d708_1540_4e39_b75d_cfcc8395a5f8.slice/crio-880feaff4fc57af0366298db579e221420ea40c4897cf9f22dc34bded6fc2e5e WatchSource:0}: Error finding container 880feaff4fc57af0366298db579e221420ea40c4897cf9f22dc34bded6fc2e5e: Status 404 returned error can't find the container with id 880feaff4fc57af0366298db579e221420ea40c4897cf9f22dc34bded6fc2e5e
Sep 17 17:35:17.979306 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:17.979235    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8538cafd_8176_42c0_bbe3_4370a998174e.slice/crio-331336cd4d54ac70b5a04733ee44802cb0aec3dd1a5322b8484ad1dc8aec6723 WatchSource:0}: Error finding container 331336cd4d54ac70b5a04733ee44802cb0aec3dd1a5322b8484ad1dc8aec6723: Status 404 returned error can't find the container with id 331336cd4d54ac70b5a04733ee44802cb0aec3dd1a5322b8484ad1dc8aec6723
Sep 17 17:35:17.981689 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:17.981660    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecef2930_3fcc_4dbb_b497_f58796174663.slice/crio-6a1e96ed08b333d32862e0ad5952da7840a31b608fe59720ca61de0cc87ab2df WatchSource:0}: Error finding container 6a1e96ed08b333d32862e0ad5952da7840a31b608fe59720ca61de0cc87ab2df: Status 404 returned error can't find the container with id 6a1e96ed08b333d32862e0ad5952da7840a31b608fe59720ca61de0cc87ab2df
Sep 17 17:35:17.983097 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:17.982657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db84866_0cdb_4b78_b53d_5c019281e212.slice/crio-d93595b2e126eb656cece77d8721262b5c6886f4b6a52224c64850ebb1fa7fd7 WatchSource:0}: Error finding container d93595b2e126eb656cece77d8721262b5c6886f4b6a52224c64850ebb1fa7fd7: Status 404 returned error can't find the container with id d93595b2e126eb656cece77d8721262b5c6886f4b6a52224c64850ebb1fa7fd7
Sep 17 17:35:17.983672 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:17.983648    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3921a5cf_6472_434c_a537_f1658c67a4d1.slice/crio-96eba0f1f80a9d2f981da926e7055043d5dd6cca338e336ee3e84f7b7c58b731 WatchSource:0}: Error finding container 96eba0f1f80a9d2f981da926e7055043d5dd6cca338e336ee3e84f7b7c58b731: Status 404 returned error can't find the container with id 96eba0f1f80a9d2f981da926e7055043d5dd6cca338e336ee3e84f7b7c58b731
Sep 17 17:35:18.425444 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.425400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:18.426006 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.425669    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:18.426006 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.425752    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:20.425729219 +0000 UTC m=+5.953511032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:18.526561 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.526462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:18.526738 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.526644    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:18.526738 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.526714    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:20.526692858 +0000 UTC m=+6.054474679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:18.627326 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.627285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:18.627528 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.627478    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:18.627528 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.627508    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:18.627528 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.627522    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:18.627702 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.627592    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:20.627570642 +0000 UTC m=+6.155352468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:18.936869 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.936818    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:30:17 +0000 UTC" deadline="2028-06-23 18:51:15.957967962 +0000 UTC"
Sep 17 17:35:18.936869 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.936867    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15481h15m57.021105849s"
Sep 17 17:35:18.954611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.954580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:18.954740 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:18.954713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:18.986620 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.986590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal" event={"ID":"a8870edaf0c4a8354936a672f9fccc76","Type":"ContainerStarted","Data":"a32a0e5721703e531bae85e031f97db5dddbc07be7387673406f784cbaac8264"}
Sep 17 17:35:18.991353 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.991327    2563 generic.go:356] "Generic (PLEG): container finished" podID="a060ffcaba569572d8e2fac80d1651df" containerID="02c62a27094a885b8e4dbcea65db907222b9763fbe935342891477010d3c7bd8" exitCode=0
Sep 17 17:35:18.991454 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.991408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal" event={"ID":"a060ffcaba569572d8e2fac80d1651df","Type":"ContainerDied","Data":"02c62a27094a885b8e4dbcea65db907222b9763fbe935342891477010d3c7bd8"}
Sep 17 17:35:18.998542 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:18.998513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-chk69" event={"ID":"4d3b541d-198a-49f5-be28-85d0ea5f1c88","Type":"ContainerStarted","Data":"bd945f979841762d4fb79403568149e170825a0a4c5e2b47e22640aa62201f34"}
Sep 17 17:35:19.007290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.007238    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-202.ec2.internal" podStartSLOduration=3.007220148 podStartE2EDuration="3.007220148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:19.006881458 +0000 UTC m=+4.534663291" watchObservedRunningTime="2026-09-17 17:35:19.007220148 +0000 UTC m=+4.535001979"
Sep 17 17:35:19.012373 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.012305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerStarted","Data":"d3cbe362dbd6b423b4e4fc971d17da9775aa32700eb2f30ed4d68a4dc2c3c459"}
Sep 17 17:35:19.022086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.022057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bxb95" event={"ID":"3921a5cf-6472-434c-a537-f1658c67a4d1","Type":"ContainerStarted","Data":"96eba0f1f80a9d2f981da926e7055043d5dd6cca338e336ee3e84f7b7c58b731"}
Sep 17 17:35:19.028097 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.028072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lwf8b" event={"ID":"6db84866-0cdb-4b78-b53d-5c019281e212","Type":"ContainerStarted","Data":"d93595b2e126eb656cece77d8721262b5c6886f4b6a52224c64850ebb1fa7fd7"}
Sep 17 17:35:19.042640 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.040972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn" event={"ID":"8538cafd-8176-42c0-bbe3-4370a998174e","Type":"ContainerStarted","Data":"331336cd4d54ac70b5a04733ee44802cb0aec3dd1a5322b8484ad1dc8aec6723"}
Sep 17 17:35:19.055965 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.055889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnnpz" event={"ID":"0df6d708-1540-4e39-b75d-cfcc8395a5f8","Type":"ContainerStarted","Data":"880feaff4fc57af0366298db579e221420ea40c4897cf9f22dc34bded6fc2e5e"}
Sep 17 17:35:19.069375 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.069277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-btktp" event={"ID":"1aacb1a6-3867-4f18-8241-4d9e294d8db0","Type":"ContainerStarted","Data":"cc68b58a9b31de2fceb4590f8821589c923205ecc962354a50cbd93f252b852b"}
Sep 17 17:35:19.096046 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.095938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"6a1e96ed08b333d32862e0ad5952da7840a31b608fe59720ca61de0cc87ab2df"}
Sep 17 17:35:19.098548 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.098517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5jhz" event={"ID":"4338b764-2fc9-42cf-a3a2-61f47006dbbd","Type":"ContainerStarted","Data":"7198864ebf2db2c8b028a2535d4730ae2083c36e3a9b7b203cd3883f7e0ffc8e"}
Sep 17 17:35:19.952917 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.952663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:19.953348 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:19.953029    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:19.953409 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:19.952888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:19.953545 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:19.953519    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:20.113960 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:20.113921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal" event={"ID":"a060ffcaba569572d8e2fac80d1651df","Type":"ContainerStarted","Data":"70bc0fa2608db7937f1a2b9f66c082971131228ffe7a469305f4eafca2dc903f"}
Sep 17 17:35:20.444341 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:20.444304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:20.444532 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.444489    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:20.444597 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.444553    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.444535383 +0000 UTC m=+9.972317201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:20.545578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:20.545535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:20.545826 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.545763    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:20.545907 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.545844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.545823203 +0000 UTC m=+10.073605021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:20.646815 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:20.646188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:20.646815 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.646373    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:20.646815 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.646395    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:20.646815 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.646407    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:20.646815 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.646465    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.646446542 +0000 UTC m=+10.174228360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:20.953361 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:20.952835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:20.953361 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:20.952962    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:21.953565 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:21.953110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:21.953565 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:21.953238    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:21.953565 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:21.953309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:21.953565 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:21.953418    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:22.955481 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:22.955003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:22.955481 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:22.955127    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:23.952930 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:23.952893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:23.953112 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:23.953033    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:23.953112 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:23.953070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:23.953224 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:23.953199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:24.479750 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:24.479714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:24.480230 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.479886    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:24.480230 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.479970    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:32.479950802 +0000 UTC m=+18.007732622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:24.581351 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:24.580856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:24.581351 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.581055    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:24.581351 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.581113    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:32.58109524 +0000 UTC m=+18.108877059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:24.681465 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:24.681436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:24.681666 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.681644    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:24.681725 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.681673    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:24.681725 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.681686    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:24.681854 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.681749    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:32.681728541 +0000 UTC m=+18.209510345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:24.953980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:24.953493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:24.953980 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:24.953600    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:25.952526 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:25.952407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:25.952526 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:25.952434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:25.953112 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:25.952554    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:25.953112 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:25.952975    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:26.952773 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:26.952741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:26.953241 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:26.952885    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:27.952980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:27.952946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:27.952980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:27.952964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:27.953392 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:27.953074    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:27.953392 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:27.953216    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:28.952538 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:28.952505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:28.952718 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:28.952612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:29.952460 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:29.952428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:29.952893 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:29.952428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:29.952893 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:29.952572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:29.952893 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:29.952669    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:30.953029 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:30.952988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:30.953471 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:30.953098    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:31.953007 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:31.952974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:31.953206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:31.952987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:31.953206 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:31.953113    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:31.953206 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:31.953185    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:32.535321 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:32.535284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:32.535489 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.535439    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:32.535532 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.535503    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:48.535484937 +0000 UTC m=+34.063267031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:32.636053 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:32.636018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:32.636205 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.636163    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:32.636261 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.636234    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:48.636213758 +0000 UTC m=+34.163995581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:32.736757 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:32.736727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:32.736926 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.736910    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:32.736981 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.736935    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:32.736981 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.736949    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:32.737072 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.737004    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:48.736985952 +0000 UTC m=+34.264767773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:32.953115 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:32.953084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:32.953294 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:32.953196    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:33.952337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:33.952301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:33.952337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:33.952330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:33.952577 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:33.952427    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:33.952577 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:33.952560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:34.952808 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:34.952766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:34.953226 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:34.952876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:35.952535 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:35.952380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:35.952628 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:35.952404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:35.952667 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:35.952623    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:35.952750 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:35.952728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:36.147375 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.147346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-btktp" event={"ID":"1aacb1a6-3867-4f18-8241-4d9e294d8db0","Type":"ContainerStarted","Data":"68869c657e063f8b3dea0cada56ee01671fcf033b338deb1bc1bcbd4f8761c55"}
Sep 17 17:35:36.149833 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.149808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"51eec1fd8095c992548047cbd8cfe3aca9110ec099507aec7dce51c3eb116c09"}
Sep 17 17:35:36.149945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.149885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"35c0a6a982b51af6dd7881e479a2711a81429c60d68e401b4cde1bd2616c5d72"}
Sep 17 17:35:36.149945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.149938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"8cd009957547217b605ca9e54e3bcb0ac882158bacd66ce85661903474c0c51c"}
Sep 17 17:35:36.150045 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.149985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"db960fd7e6791d16675c71b66c2e88f438cf4d345a99cadc29327d50d9e58199"}
Sep 17 17:35:36.151279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.151254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t5jhz" event={"ID":"4338b764-2fc9-42cf-a3a2-61f47006dbbd","Type":"ContainerStarted","Data":"876338f6c6d4b829cead11885298ba8be8030794c4c6e5388fd091bfe89be505"}
Sep 17 17:35:36.152728 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.152707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-chk69" event={"ID":"4d3b541d-198a-49f5-be28-85d0ea5f1c88","Type":"ContainerStarted","Data":"fe5b4d3bd82aba64acca419370ded9274c89ff7e70855c2d404df6ac5acda927"}
Sep 17 17:35:36.154284 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.154255    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5c05eb-c75a-45ad-8f2a-06ea33b0005d" containerID="52e687905a1318f6c5003c2725d628453e41170a64afab0199f34a45981d576c" exitCode=0
Sep 17 17:35:36.154390 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.154335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerDied","Data":"52e687905a1318f6c5003c2725d628453e41170a64afab0199f34a45981d576c"}
Sep 17 17:35:36.159381 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.159255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lwf8b" event={"ID":"6db84866-0cdb-4b78-b53d-5c019281e212","Type":"ContainerStarted","Data":"51793989e41d764dca12aeacb3c6cc11785adf1c5618c17f7eba7812afc2ae6c"}
Sep 17 17:35:36.160824 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.160782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn" event={"ID":"8538cafd-8176-42c0-bbe3-4370a998174e","Type":"ContainerStarted","Data":"fc999e9ecdfcc43aa9f5342ffd44e4f4b7d687b67b994008a0622213669246a4"}
Sep 17 17:35:36.162348 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.162325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnnpz" event={"ID":"0df6d708-1540-4e39-b75d-cfcc8395a5f8","Type":"ContainerStarted","Data":"d706e7dde52c395a77ab7cd343c0d93415fdb2ac4d2d6076b0049114f0400a8e"}
Sep 17 17:35:36.164902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.164860    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-202.ec2.internal" podStartSLOduration=20.164844749 podStartE2EDuration="20.164844749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:20.130995231 +0000 UTC m=+5.658777062" watchObservedRunningTime="2026-09-17 17:35:36.164844749 +0000 UTC m=+21.692626575"
Sep 17 17:35:36.181034 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.180989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-btktp" podStartSLOduration=7.143168246 podStartE2EDuration="21.180971397s" totalImagesPullingTime="14.037803151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.164345023 +0000 UTC m=+21.692126860" watchObservedRunningTime="2026-09-17 17:35:36.180971397 +0000 UTC m=+21.708753221"
Sep 17 17:35:36.181275 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.181246    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t5jhz" podStartSLOduration=3.59406857 podStartE2EDuration="21.181236739s" totalImagesPullingTime="17.587168169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.181077182 +0000 UTC m=+21.708859007" watchObservedRunningTime="2026-09-17 17:35:36.181236739 +0000 UTC m=+21.709018566"
Sep 17 17:35:36.202805 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.202761    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tnnpz" podStartSLOduration=3.615699618 podStartE2EDuration="21.20275046s" totalImagesPullingTime="17.587050842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.202417633 +0000 UTC m=+21.730199460" watchObservedRunningTime="2026-09-17 17:35:36.20275046 +0000 UTC m=+21.730532285"
Sep 17 17:35:36.254604 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.254560    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-chk69" podStartSLOduration=3.614420624 podStartE2EDuration="21.254546417s" totalImagesPullingTime="17.640125793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.254050944 +0000 UTC m=+21.781832769" watchObservedRunningTime="2026-09-17 17:35:36.254546417 +0000 UTC m=+21.782328246"
Sep 17 17:35:36.271396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.271194    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lwf8b" podStartSLOduration=3.690372527 podStartE2EDuration="21.271181966s" totalImagesPullingTime="17.580809439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.271124189 +0000 UTC m=+21.798906015" watchObservedRunningTime="2026-09-17 17:35:36.271181966 +0000 UTC m=+21.798963792"
Sep 17 17:35:36.359394 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.359373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:36.360062 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.360041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:36.845707 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.845685    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:35:36.863886 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.863732    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:35:36.845705646Z","UUID":"b83f2bd6-3f05-479f-9321-0ca239529868","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:35:36.865421 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.865402    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:35:36.865527 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.865436    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:35:36.953164 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:36.953130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:36.953309 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:36.953227    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:37.167034 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.166950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bxb95" event={"ID":"3921a5cf-6472-434c-a537-f1658c67a4d1","Type":"ContainerStarted","Data":"6bf32839735d6a8d36600a57eb5e8bfb20b1fa5a6b201894905102e6433bcf2a"}
Sep 17 17:35:37.169163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.169123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn" event={"ID":"8538cafd-8176-42c0-bbe3-4370a998174e","Type":"ContainerStarted","Data":"fdf7395b0e6ee90f7d33398f139452131d2c63e6e40a13cbb165479f96a2f305"}
Sep 17 17:35:37.172705 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.172682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"6a689d9b6911350fc4f0e895b9559947775f2210aa799a6fe5041749b95aba20"}
Sep 17 17:35:37.172825 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.172761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"72f33e1e2d1515c2199e479aa5326d6b3b2044f64374320e0b04534fe576121c"}
Sep 17 17:35:37.185258 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.185207    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bxb95" podStartSLOduration=4.60800222 podStartE2EDuration="22.185183094s" totalImagesPullingTime="17.577180874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:37.184162912 +0000 UTC m=+22.711944739" watchObservedRunningTime="2026-09-17 17:35:37.185183094 +0000 UTC m=+22.712964920"
Sep 17 17:35:37.952879 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.952852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:37.952975 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:37.952895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:37.953026 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:37.952981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:37.953134 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:37.953111    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:38.177074 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:38.177036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn" event={"ID":"8538cafd-8176-42c0-bbe3-4370a998174e","Type":"ContainerStarted","Data":"14851b5a9d342796b292a494e83fbd0c17649c606a2f0bba212cd3ffe13cc553"}
Sep 17 17:35:38.177549 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:38.177343    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:35:38.221529 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:38.221472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9plzn" podStartSLOduration=3.6940578459999998 podStartE2EDuration="23.221449629s" totalImagesPullingTime="19.527391783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:38.220841109 +0000 UTC m=+23.748622938" watchObservedRunningTime="2026-09-17 17:35:38.221449629 +0000 UTC m=+23.749231462"
Sep 17 17:35:38.952302 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:38.952273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:38.952480 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:38.952395    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:39.183563 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:39.183534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"b0271a6a1f8ad271740990c13ed12c29caeb73da9c2a1c2217be6e71bb409f03"}
Sep 17 17:35:39.952860 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:39.952833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:39.952860 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:39.952861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:39.953079 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:39.952963    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:39.953119 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:39.953098    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:40.952821 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:40.952601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:40.953468 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:40.952861    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:41.189448 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.189413    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5c05eb-c75a-45ad-8f2a-06ea33b0005d" containerID="ca52f87ac5ff6e0d10443495bee0f0ae7d935b6fe61f1e2b6d570e02dfc59c25" exitCode=0
Sep 17 17:35:41.189626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.189501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerDied","Data":"ca52f87ac5ff6e0d10443495bee0f0ae7d935b6fe61f1e2b6d570e02dfc59c25"}
Sep 17 17:35:41.193401 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.193378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" event={"ID":"ecef2930-3fcc-4dbb-b497-f58796174663","Type":"ContainerStarted","Data":"bf39f07a1125689515e7dbeaadfdc51ef2fc6ac00d7664f387a3f891d2fb1184"}
Sep 17 17:35:41.193664 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.193640    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:41.210407 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.210351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:41.249804 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.249753    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8" podStartSLOduration=8.598041127 podStartE2EDuration="26.249728848s" totalImagesPullingTime="17.651687721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:41.248457592 +0000 UTC m=+26.776239419" watchObservedRunningTime="2026-09-17 17:35:41.249728848 +0000 UTC m=+26.777510673"
Sep 17 17:35:41.952612 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.952448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:41.952836 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:41.952506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:41.952836 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:41.952698    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:41.952836 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:41.952753    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:42.197845 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.197812    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5c05eb-c75a-45ad-8f2a-06ea33b0005d" containerID="758df33ae739e562b11a963a21140a6b1f9083c3102bf83b264192c9c022bd6f" exitCode=0
Sep 17 17:35:42.198019 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.197853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerDied","Data":"758df33ae739e562b11a963a21140a6b1f9083c3102bf83b264192c9c022bd6f"}
Sep 17 17:35:42.198329 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.198310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:42.198462 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.198387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:42.213160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.213142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:35:42.513421 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.513348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sd6ff"]
Sep 17 17:35:42.513588 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.513485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:42.513588 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:42.513562    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:42.517980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.517954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zxjft"]
Sep 17 17:35:42.518083 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.518053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:42.518140 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:42.518125    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:42.521896 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.521874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7swdh"]
Sep 17 17:35:42.521998 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:42.521953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:42.522081 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:42.522013    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:43.202142 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.202109    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5c05eb-c75a-45ad-8f2a-06ea33b0005d" containerID="5f0ab6348d24509394329e91bc0d4759dc3ec5e86816100a948b93fac41acae3" exitCode=0
Sep 17 17:35:43.202505 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.202196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerDied","Data":"5f0ab6348d24509394329e91bc0d4759dc3ec5e86816100a948b93fac41acae3"}
Sep 17 17:35:43.676898 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.676859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:43.822913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.822817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:43.952129 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.952095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:43.952302 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.952201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:43.952373 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:43.952328    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:43.952373 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:43.952209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:43.952373 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:43.952206    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:43.952532 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:43.952450    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:45.658441 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.658159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:45.658912 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.658522    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:35:45.659007 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.658979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-btktp"
Sep 17 17:35:45.740949 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.740905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:45.952649 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.952620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:45.952848 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.952734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:45.952848 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:45.952735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:45.952848 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:45.952736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:45.953029 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:45.952857    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:45.953029 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:45.952918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:46.750668 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:46.750634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:46.833650 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:46.833612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:47.338140 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:47.338101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:47.435815 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:47.435774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:47.953067 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:47.953037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:47.953524 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:47.953037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:47.953524 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:47.953131    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7swdh" podUID="b206d4e1-1b71-457b-b416-a6d0dfdf36a2"
Sep 17 17:35:47.953524 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:47.953199    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sd6ff" podUID="05a68412-83e9-486b-a686-65730e2448ba"
Sep 17 17:35:47.953524 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:47.953037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:47.953524 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:47.953308    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:35:48.217081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.217011    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5c05eb-c75a-45ad-8f2a-06ea33b0005d" containerID="f67ceb67a9a5162a974d35ed9be7c302d6a85cab6284feebb3e50a685e1a983d" exitCode=0
Sep 17 17:35:48.217081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.217067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerDied","Data":"f67ceb67a9a5162a974d35ed9be7c302d6a85cab6284feebb3e50a685e1a983d"}
Sep 17 17:35:48.247713 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.247691    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-202.ec2.internal" event="NodeReady"
Sep 17 17:35:48.247837 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.247831    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:35:48.289044 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.289011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b9c5577bd-tm4nb"]
Sep 17 17:35:48.309039 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.309018    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jczs6"]
Sep 17 17:35:48.309156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.309138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.311701 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.311683    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:35:48.311853 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.311736    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:35:48.311920 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.311870    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l58cx\""
Sep 17 17:35:48.319086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.319069    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:35:48.321107 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.321087    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:35:48.329631 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.329605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.329631 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.329619    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zmsks"]
Sep 17 17:35:48.332638 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.332609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:35:48.332724 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.332658    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:35:48.332845 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.332826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:35:48.333341 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.333209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2pzr6\""
Sep 17 17:35:48.347841 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.347813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9c5577bd-tm4nb"]
Sep 17 17:35:48.347933 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.347849    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jczs6"]
Sep 17 17:35:48.347933 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.347869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmsks"]
Sep 17 17:35:48.347933 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.347876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.350162 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.350143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:35:48.350238 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.350181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:35:48.350393 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.350381    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksc59\""
Sep 17 17:35:48.442298 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.442270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:48.454295 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7af89-b3ff-44d8-8354-525579dcdf5b-config-volume\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.454388 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbv8\" (UniqueName: \"kubernetes.io/projected/9db7af89-b3ff-44d8-8354-525579dcdf5b-kube-api-access-fmbv8\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.454388 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-ca-trust-extracted\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454499 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-bound-sa-token\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454499 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-image-registry-private-configuration\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454499 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-installation-pull-secrets\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxqp\" (UniqueName: \"kubernetes.io/projected/4622bbec-ed20-4923-a475-1e138ed7b050-kube-api-access-vhxqp\") pod \"ingress-canary-jczs6\" (UID: \"4622bbec-ed20-4923-a475-1e138ed7b050\") " pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.454644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-trusted-ca\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db7af89-b3ff-44d8-8354-525579dcdf5b-tmp-dir\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.454644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-certificates\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccs7\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-kube-api-access-gccs7\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454879 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4622bbec-ed20-4923-a475-1e138ed7b050-cert\") pod \"ingress-canary-jczs6\" (UID: \"4622bbec-ed20-4923-a475-1e138ed7b050\") " pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.454879 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.454879 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.454720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.515104 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.515049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:48.556010 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.555985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-certificates\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556101 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccs7\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-kube-api-access-gccs7\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556101 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4622bbec-ed20-4923-a475-1e138ed7b050-cert\") pod \"ingress-canary-jczs6\" (UID: \"4622bbec-ed20-4923-a475-1e138ed7b050\") " pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.556101 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:48.556234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.556234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7af89-b3ff-44d8-8354-525579dcdf5b-config-volume\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.556234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbv8\" (UniqueName: \"kubernetes.io/projected/9db7af89-b3ff-44d8-8354-525579dcdf5b-kube-api-access-fmbv8\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.556234 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556148    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556240    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556249    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556311    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:49.056293758 +0000 UTC m=+34.584075577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-ca-trust-extracted\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-bound-sa-token\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-image-registry-private-configuration\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556460 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556225    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-installation-pull-secrets\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556503    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.55648822 +0000 UTC m=+66.084270038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxqp\" (UniqueName: \"kubernetes.io/projected/4622bbec-ed20-4923-a475-1e138ed7b050-kube-api-access-vhxqp\") pod \"ingress-canary-jczs6\" (UID: \"4622bbec-ed20-4923-a475-1e138ed7b050\") " pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-ca-trust-extracted\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.556601    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:35:49.056586177 +0000 UTC m=+34.584367998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-trusted-ca\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db7af89-b3ff-44d8-8354-525579dcdf5b-tmp-dir\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-certificates\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.556827 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9db7af89-b3ff-44d8-8354-525579dcdf5b-config-volume\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.557148 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.556982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db7af89-b3ff-44d8-8354-525579dcdf5b-tmp-dir\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.557978 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.557960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-trusted-ca\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.560139 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.560122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4622bbec-ed20-4923-a475-1e138ed7b050-cert\") pod \"ingress-canary-jczs6\" (UID: \"4622bbec-ed20-4923-a475-1e138ed7b050\") " pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.560239 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.560162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-image-registry-private-configuration\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.560239 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.560192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-installation-pull-secrets\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.565606 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.565565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxqp\" (UniqueName: \"kubernetes.io/projected/4622bbec-ed20-4923-a475-1e138ed7b050-kube-api-access-vhxqp\") pod \"ingress-canary-jczs6\" (UID: \"4622bbec-ed20-4923-a475-1e138ed7b050\") " pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.565893 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.565874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-bound-sa-token\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.566008 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.565915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbv8\" (UniqueName: \"kubernetes.io/projected/9db7af89-b3ff-44d8-8354-525579dcdf5b-kube-api-access-fmbv8\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:48.566172 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.566154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccs7\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-kube-api-access-gccs7\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:48.638583 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.638564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jczs6"
Sep 17 17:35:48.657970 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.657948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:48.658097 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.658080    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:48.658175 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.658135    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret podName:05a68412-83e9-486b-a686-65730e2448ba nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.658118961 +0000 UTC m=+66.185900766 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret") pod "global-pull-secret-syncer-sd6ff" (UID: "05a68412-83e9-486b-a686-65730e2448ba") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:48.759487 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.759359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:48.759639 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.759521    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:48.759639 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.759553    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:48.759639 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.759569    2563 projected.go:196] Error preparing data for projected volume kube-api-access-zkqnl for pod openshift-network-diagnostics/network-check-target-7swdh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:48.759639 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:48.759638    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl podName:b206d4e1-1b71-457b-b416-a6d0dfdf36a2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.759619682 +0000 UTC m=+66.287401488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zkqnl" (UniqueName: "kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl") pod "network-check-target-7swdh" (UID: "b206d4e1-1b71-457b-b416-a6d0dfdf36a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:48.895245 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.895153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:48.966931 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:48.966906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jczs6"]
Sep 17 17:35:48.970142 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:35:48.970119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4622bbec_ed20_4923_a475_1e138ed7b050.slice/crio-5097572021eb14129606624ce121c119126796de00a6f16671a0962e2f4bd08c WatchSource:0}: Error finding container 5097572021eb14129606624ce121c119126796de00a6f16671a0962e2f4bd08c: Status 404 returned error can't find the container with id 5097572021eb14129606624ce121c119126796de00a6f16671a0962e2f4bd08c
Sep 17 17:35:49.062033 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.062011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:49.062157 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.062049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:49.062201 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:49.062158    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:49.062201 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:49.062175    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:35:49.062276 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:49.062221    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:49.062276 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:49.062228    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:50.062213661 +0000 UTC m=+35.589995465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:35:49.062276 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:49.062258    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:35:50.062246672 +0000 UTC m=+35.590028475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:49.225951 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.225919    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5c05eb-c75a-45ad-8f2a-06ea33b0005d" containerID="ce9c81e0361ab68f33ecdba846e55c4d32e46574f60cd69a6c3923b9cf0690f2" exitCode=0
Sep 17 17:35:49.226124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.226000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerDied","Data":"ce9c81e0361ab68f33ecdba846e55c4d32e46574f60cd69a6c3923b9cf0690f2"}
Sep 17 17:35:49.227267 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.227247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jczs6" event={"ID":"4622bbec-ed20-4923-a475-1e138ed7b050","Type":"ContainerStarted","Data":"5097572021eb14129606624ce121c119126796de00a6f16671a0962e2f4bd08c"}
Sep 17 17:35:49.952448 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.952418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:35:49.952448 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.952446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:35:49.952670 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.952617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:35:49.956158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.956133    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:49.956296 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.956281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:49.957347 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.957175    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6xmz\""
Sep 17 17:35:49.957347 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.957202    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:49.957513 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.957430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wr2c\""
Sep 17 17:35:49.957513 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:49.957436    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:50.071807 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:50.071759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:50.072273 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:50.071843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:50.072273 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:50.071903    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:50.072273 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:50.071922    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:35:50.072273 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:50.071978    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:52.071962972 +0000 UTC m=+37.599744776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:35:50.072273 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:50.072014    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:50.072273 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:50.072075    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:35:52.072057566 +0000 UTC m=+37.599839372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:50.233443 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:50.233363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knj9c" event={"ID":"2d5c05eb-c75a-45ad-8f2a-06ea33b0005d","Type":"ContainerStarted","Data":"43421bcccb847827c35ba5403f7a28b9e305728232e970365a8964d98522c5c3"}
Sep 17 17:35:50.482033 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:50.481991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:50.649386 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:50.649299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:52.084921 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:52.084755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:52.085208 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:52.084972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:52.085208 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:52.084904    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:52.085208 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:52.085047    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:52.085208 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:52.085059    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:35:52.085208 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:52.085049    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:35:56.08503391 +0000 UTC m=+41.612815732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:52.085208 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:52.085100    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:56.085089868 +0000 UTC m=+41.612871672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:35:52.239355 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:52.239322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jczs6" event={"ID":"4622bbec-ed20-4923-a475-1e138ed7b050","Type":"ContainerStarted","Data":"d9089b6aa4d5afa556ce2e055abf270357b5ff736c2ce9d2df2f854d3b912d7a"}
Sep 17 17:35:52.267773 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:52.267731    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-knj9c" podStartSLOduration=10.013179016 podStartE2EDuration="37.267716995s" totalImagesPullingTime="27.254537979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:50.272903671 +0000 UTC m=+35.800685497" watchObservedRunningTime="2026-09-17 17:35:52.267716995 +0000 UTC m=+37.795498820"
Sep 17 17:35:52.268349 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:52.268316    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jczs6" podStartSLOduration=1.766315058 podStartE2EDuration="4.268305823s" totalImagesPullingTime="2.501990765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:52.26685424 +0000 UTC m=+37.794636067" watchObservedRunningTime="2026-09-17 17:35:52.268305823 +0000 UTC m=+37.796087648"
Sep 17 17:35:54.336142 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:54.336106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:54.426230 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:54.426199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:55.440658 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:55.440619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:55.540886 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:55.540857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:56.114088 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:56.114053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:35:56.114249 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:35:56.114139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:35:56.114249 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:56.114197    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:56.114249 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:56.114226    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:56.114249 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:56.114237    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:35:56.114377 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:56.114260    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:36:04.114243996 +0000 UTC m=+49.642025801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:56.114377 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:35:56.114278    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:04.114266799 +0000 UTC m=+49.642048613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:36:04.171679 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:04.171649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:36:04.172103 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:04.171687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:36:04.172103 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:04.171804    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:04.172103 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:04.171823    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:36:04.172103 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:04.171852    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:04.172103 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:04.171873    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.17185784 +0000 UTC m=+65.699639658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:36:04.172103 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:04.171890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:36:20.171879683 +0000 UTC m=+65.699661487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:08.697886 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:08.697851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:08.768775 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:08.768743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:08.881190 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:08.881146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:08.961483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:08.961401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:14.215310 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:14.215281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmbb8"
Sep 17 17:36:14.382716 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:14.382687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:14.459503 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:14.459457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:16.268246 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:16.268204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:16.388452 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:16.388425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:16.525245 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:16.525155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:20.180754 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.180715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:36:20.181148 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.180766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:36:20.181148 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.180884    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:20.181148 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.180895    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:20.181148 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.180903    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:36:20.181148 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.180948    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:36:52.180935518 +0000 UTC m=+97.708717322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:20.181148 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.180961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:52.180955796 +0000 UTC m=+97.708737600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:36:20.583561 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.583460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:36:20.586145 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.586126    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:36:20.594229 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.594212    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:36:20.594302 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:20.594263    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:24.594249297 +0000 UTC m=+130.122031101 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : secret "metrics-daemon-secret" not found
Sep 17 17:36:20.684392 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.684364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:36:20.687874 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.687853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:36:20.698648 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.698620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05a68412-83e9-486b-a686-65730e2448ba-original-pull-secret\") pod \"global-pull-secret-syncer-sd6ff\" (UID: \"05a68412-83e9-486b-a686-65730e2448ba\") " pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:36:20.784986 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.784960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:36:20.789290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.789271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:36:20.798293 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.798272    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:36:20.808704 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.808679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqnl\" (UniqueName: \"kubernetes.io/projected/b206d4e1-1b71-457b-b416-a6d0dfdf36a2-kube-api-access-zkqnl\") pod \"network-check-target-7swdh\" (UID: \"b206d4e1-1b71-457b-b416-a6d0dfdf36a2\") " pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:36:20.867029 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.866974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6xmz\""
Sep 17 17:36:20.874867 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.874852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:36:20.878454 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:20.878440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sd6ff"
Sep 17 17:36:21.158602 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:21.158430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sd6ff"]
Sep 17 17:36:21.161143 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:36:21.161115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a68412_83e9_486b_a686_65730e2448ba.slice/crio-8ddf01b1da09483e1f1c6c9feb984327f8e42b058455a0fdb638fcd7d6f9781d WatchSource:0}: Error finding container 8ddf01b1da09483e1f1c6c9feb984327f8e42b058455a0fdb638fcd7d6f9781d: Status 404 returned error can't find the container with id 8ddf01b1da09483e1f1c6c9feb984327f8e42b058455a0fdb638fcd7d6f9781d
Sep 17 17:36:21.161330 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:21.161312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7swdh"]
Sep 17 17:36:21.164071 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:36:21.164055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb206d4e1_1b71_457b_b416_a6d0dfdf36a2.slice/crio-ed0d8e5254f86bb085302b6030cdf4da6b062d999333d26e0bb162ac8941cb0d WatchSource:0}: Error finding container ed0d8e5254f86bb085302b6030cdf4da6b062d999333d26e0bb162ac8941cb0d: Status 404 returned error can't find the container with id ed0d8e5254f86bb085302b6030cdf4da6b062d999333d26e0bb162ac8941cb0d
Sep 17 17:36:21.310300 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:21.310268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sd6ff" event={"ID":"05a68412-83e9-486b-a686-65730e2448ba","Type":"ContainerStarted","Data":"8ddf01b1da09483e1f1c6c9feb984327f8e42b058455a0fdb638fcd7d6f9781d"}
Sep 17 17:36:21.311391 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:21.311361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7swdh" event={"ID":"b206d4e1-1b71-457b-b416-a6d0dfdf36a2","Type":"ContainerStarted","Data":"ed0d8e5254f86bb085302b6030cdf4da6b062d999333d26e0bb162ac8941cb0d"}
Sep 17 17:36:26.327075 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:26.327036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sd6ff" event={"ID":"05a68412-83e9-486b-a686-65730e2448ba","Type":"ContainerStarted","Data":"11158ad2e4c329614d813ecdb4c16e0efac1ef66ad16ad21525216554f8a49a3"}
Sep 17 17:36:26.328551 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:26.328528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7swdh" event={"ID":"b206d4e1-1b71-457b-b416-a6d0dfdf36a2","Type":"ContainerStarted","Data":"9eca9e6f9bb2e2f23d1e051e34410813203ed38b48e305d74d0535cd6e1f8e7a"}
Sep 17 17:36:26.328650 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:26.328586    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:36:26.355392 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:26.355348    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sd6ff" podStartSLOduration=67.15539864 podStartE2EDuration="1m11.355334852s" totalImagesPullingTime="4.199936212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:26.354877341 +0000 UTC m=+71.882659168" watchObservedRunningTime="2026-09-17 17:36:26.355334852 +0000 UTC m=+71.883116678"
Sep 17 17:36:26.372418 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:26.372379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7swdh" podStartSLOduration=67.177464533 podStartE2EDuration="1m11.372369274s" totalImagesPullingTime="4.194904741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:26.371858718 +0000 UTC m=+71.899640544" watchObservedRunningTime="2026-09-17 17:36:26.372369274 +0000 UTC m=+71.900151096"
Sep 17 17:36:52.205651 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:52.205618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:36:52.206094 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:52.205668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:36:52.206094 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:52.205770    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:52.206094 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:52.205787    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b9c5577bd-tm4nb: secret "image-registry-tls" not found
Sep 17 17:36:52.206094 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:52.205809    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:52.206094 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:52.205857    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls podName:3bb44e32-45d0-4fde-b7cf-0384ab5f1686 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:56.205841341 +0000 UTC m=+161.733623145 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls") pod "image-registry-6b9c5577bd-tm4nb" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686") : secret "image-registry-tls" not found
Sep 17 17:36:52.206094 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:36:52.205871    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls podName:9db7af89-b3ff-44d8-8354-525579dcdf5b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:56.205864759 +0000 UTC m=+161.733646562 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls") pod "dns-default-zmsks" (UID: "9db7af89-b3ff-44d8-8354-525579dcdf5b") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:57.333222 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:36:57.333184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7swdh"
Sep 17 17:37:24.639425 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:24.639379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:37:24.639933 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:24.639521    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:37:24.639933 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:24.639596    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs podName:d2d139ed-fedb-42f4-b2d7-10e810b16090 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:26.63958041 +0000 UTC m=+252.167362219 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs") pod "network-metrics-daemon-zxjft" (UID: "d2d139ed-fedb-42f4-b2d7-10e810b16090") : secret "metrics-daemon-secret" not found
Sep 17 17:37:25.301609 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:25.301571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:27.708114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.708080    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56879d8b98-cgl46"]
Sep 17 17:37:27.709800 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.709771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.712975 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.712951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 17:37:27.712975 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.712967    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:37:27.713158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.712950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 17:37:27.713264 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.713248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 17:37:27.713333 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.713282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 17:37:27.713721 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.713708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bpdff\""
Sep 17 17:37:27.714494 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.714477    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:37:27.725823 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.725781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56879d8b98-cgl46"]
Sep 17 17:37:27.865939 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.865911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-default-certificate\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.866117 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.865947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.866117 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.865965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.866117 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.866021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-stats-auth\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.866117 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.866051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbnn\" (UniqueName: \"kubernetes.io/projected/6abc45d9-b9e0-4f47-9819-97ad80b332ee-kube-api-access-xjbnn\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.870409 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.870383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:27.966431 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.966350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-default-certificate\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.966431 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.966390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.966431 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.966407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.966695 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.966445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-stats-auth\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.966695 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.966494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbnn\" (UniqueName: \"kubernetes.io/projected/6abc45d9-b9e0-4f47-9819-97ad80b332ee-kube-api-access-xjbnn\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.966695 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:27.966518    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:37:27.966695 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:27.966562    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:28.466542537 +0000 UTC m=+133.994324365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : configmap references non-existent config key: service-ca.crt
Sep 17 17:37:27.966695 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:27.966585    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:28.466575033 +0000 UTC m=+133.994356837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : secret "router-metrics-certs-default" not found
Sep 17 17:37:27.967105 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.967053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:27.969087 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.969067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-stats-auth\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.969180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.969100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-default-certificate\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:27.975318 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:27.975296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbnn\" (UniqueName: \"kubernetes.io/projected/6abc45d9-b9e0-4f47-9819-97ad80b332ee-kube-api-access-xjbnn\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:28.469395 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:28.469350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:28.469395 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:28.469402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:28.469646 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:28.469547    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:37:28.469646 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:28.469554    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:29.469535401 +0000 UTC m=+134.997317308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : configmap references non-existent config key: service-ca.crt
Sep 17 17:37:28.469646 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:28.469610    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:29.469593841 +0000 UTC m=+134.997375660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : secret "router-metrics-certs-default" not found
Sep 17 17:37:29.476262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:29.476228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:29.476262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:29.476259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:29.476718 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:29.476357    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:37:29.476718 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:29.476406    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:31.476393766 +0000 UTC m=+137.004175570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : secret "router-metrics-certs-default" not found
Sep 17 17:37:29.476718 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:29.476419    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:31.476413448 +0000 UTC m=+137.004195252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : configmap references non-existent config key: service-ca.crt
Sep 17 17:37:31.490933 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.490901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:31.490933 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.490936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:31.491479 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:31.491059    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:37:31.491479 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:31.491094    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:35.491072974 +0000 UTC m=+141.018854793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : configmap references non-existent config key: service-ca.crt
Sep 17 17:37:31.491479 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:31.491116    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:35.491105436 +0000 UTC m=+141.018887253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : secret "router-metrics-certs-default" not found
Sep 17 17:37:31.812997 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.812881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:31.828036 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.827997    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pxzlk"]
Sep 17 17:37:31.830055 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.830040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:31.832544 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.832520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:37:31.833220 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.833203    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:37:31.833280 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.833214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:37:31.833329 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.833212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:37:31.833329 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.833214    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkst8\""
Sep 17 17:37:31.843069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.843045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pxzlk"]
Sep 17 17:37:31.894739 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.894706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:31.994356 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.994310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:31.994356 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.994360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xsn\" (UniqueName: \"kubernetes.io/projected/38a82660-d31e-4922-a63e-98d2ccf89243-kube-api-access-22xsn\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:31.994528 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.994455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38a82660-d31e-4922-a63e-98d2ccf89243-crio-socket\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:31.994528 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.994493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:31.994528 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:31.994516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38a82660-d31e-4922-a63e-98d2ccf89243-data-volume\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095475 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38a82660-d31e-4922-a63e-98d2ccf89243-data-volume\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095475 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095651 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xsn\" (UniqueName: \"kubernetes.io/projected/38a82660-d31e-4922-a63e-98d2ccf89243-kube-api-access-22xsn\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095651 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38a82660-d31e-4922-a63e-98d2ccf89243-crio-socket\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095651 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095651 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.095594    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.095823 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.095688    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.595651274 +0000 UTC m=+138.123433092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.095823 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.095689    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:32.095823 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38a82660-d31e-4922-a63e-98d2ccf89243-crio-socket\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.095823 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.095723    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.595714434 +0000 UTC m=+138.123496253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:32.095823 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.095729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38a82660-d31e-4922-a63e-98d2ccf89243-data-volume\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.104727 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.104702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xsn\" (UniqueName: \"kubernetes.io/projected/38a82660-d31e-4922-a63e-98d2ccf89243-kube-api-access-22xsn\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.600279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.600240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.600655 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.600259    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.600655 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:32.600338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:32.600655 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.600408    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:33.600394159 +0000 UTC m=+139.128175999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.600655 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.600437    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:32.600655 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:32.600479    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:33.600467134 +0000 UTC m=+139.128248938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:33.607487 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:33.607444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:33.607897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:33.607523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:33.607897 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:33.607571    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:33.607897 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:33.607616    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:33.607897 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:33.607640    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:35.60762464 +0000 UTC m=+141.135406467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:33.607897 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:33.607658    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:35.607651641 +0000 UTC m=+141.135433445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:33.750312 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:33.750285    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lwf8b_6db84866-0cdb-4b78-b53d-5c019281e212/dns-node-resolver/0.log"
Sep 17 17:37:34.551492 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:34.551464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5jhz_4338b764-2fc9-42cf-a3a2-61f47006dbbd/node-ca/0.log"
Sep 17 17:37:35.349632 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:35.349593    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jczs6_4622bbec-ed20-4923-a475-1e138ed7b050/serve-healthcheck-canary/0.log"
Sep 17 17:37:35.523178 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:35.523140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:35.523178 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:35.523178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:35.523430 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.523314    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:43.523296511 +0000 UTC m=+149.051078315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : configmap references non-existent config key: service-ca.crt
Sep 17 17:37:35.523430 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.523325    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:37:35.523430 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.523387    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:43.523371675 +0000 UTC m=+149.051153499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : secret "router-metrics-certs-default" not found
Sep 17 17:37:35.623883 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:35.623765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:35.624031 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.623887    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:35.624031 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.623950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:39.623935348 +0000 UTC m=+145.151717152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:35.624031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:35.623996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:35.624155 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.624079    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:35.624155 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:35.624108    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:39.624100898 +0000 UTC m=+145.151882702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:37.742629 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.742598    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-p9xpb"]
Sep 17 17:37:37.744630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.744613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.747626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.747603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:37:37.747745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.747703    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:37:37.748159 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.748134    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:37:37.748449 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.748432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:37:37.748509 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.748440    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dmrnd\""
Sep 17 17:37:37.752771 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.752748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:37:37.759694 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.759672    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-p9xpb"]
Sep 17 17:37:37.840225 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.840183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-trusted-ca\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.840396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.840244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-config\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.840396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.840290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-serving-cert\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.840396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.840354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr755\" (UniqueName: \"kubernetes.io/projected/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-kube-api-access-kr755\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.940778 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.940742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr755\" (UniqueName: \"kubernetes.io/projected/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-kube-api-access-kr755\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.940979 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.940852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-trusted-ca\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.940979 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.940894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-config\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.940979 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.940921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-serving-cert\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.941562 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.941533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-config\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.941765 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.941747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-trusted-ca\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.943168 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.943144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-serving-cert\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:37.949004 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:37.948978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr755\" (UniqueName: \"kubernetes.io/projected/4e52a62f-f2fa-4b66-be91-758b8dffcfbd-kube-api-access-kr755\") pod \"console-operator-fd4bb46c-p9xpb\" (UID: \"4e52a62f-f2fa-4b66-be91-758b8dffcfbd\") " pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:38.054930 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:38.054817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:38.324027 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:38.323950    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-p9xpb"]
Sep 17 17:37:38.327317 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:37:38.327292    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e52a62f_f2fa_4b66_be91_758b8dffcfbd.slice/crio-6d17830d0c5b2748d2fb1c0c95e6a95119efcd6853718323325c05b4685e4bb0 WatchSource:0}: Error finding container 6d17830d0c5b2748d2fb1c0c95e6a95119efcd6853718323325c05b4685e4bb0: Status 404 returned error can't find the container with id 6d17830d0c5b2748d2fb1c0c95e6a95119efcd6853718323325c05b4685e4bb0
Sep 17 17:37:38.499465 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:38.499424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" event={"ID":"4e52a62f-f2fa-4b66-be91-758b8dffcfbd","Type":"ContainerStarted","Data":"6d17830d0c5b2748d2fb1c0c95e6a95119efcd6853718323325c05b4685e4bb0"}
Sep 17 17:37:39.656181 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:39.656133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:39.656630 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:39.656246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:39.656630 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:39.656261    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:39.656630 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:39.656331    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:47.6563114 +0000 UTC m=+153.184093204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:39.656630 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:39.656373    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:39.656630 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:39.656442    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:47.656423962 +0000 UTC m=+153.184205782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:40.505990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:40.505915    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/0.log"
Sep 17 17:37:40.505990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:40.505953    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e52a62f-f2fa-4b66-be91-758b8dffcfbd" containerID="8621205ac47976c62909995856f2e17fa98b2309bfde0199a45b1cfa5352bb5a" exitCode=255
Sep 17 17:37:40.506162 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:40.506021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" event={"ID":"4e52a62f-f2fa-4b66-be91-758b8dffcfbd","Type":"ContainerDied","Data":"8621205ac47976c62909995856f2e17fa98b2309bfde0199a45b1cfa5352bb5a"}
Sep 17 17:37:40.506282 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:40.506268    2563 scope.go:117] "RemoveContainer" containerID="8621205ac47976c62909995856f2e17fa98b2309bfde0199a45b1cfa5352bb5a"
Sep 17 17:37:40.543290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:40.543237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:40.643693 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:40.643661    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.510440 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.510408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:37:41.510880 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.510862    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/0.log"
Sep 17 17:37:41.510927 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.510902    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e52a62f-f2fa-4b66-be91-758b8dffcfbd" containerID="e03a457b9aa72b8f810122d05338d07a4709b298e247a01d6e8f69b5dd640ab6" exitCode=255
Sep 17 17:37:41.510968 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.510941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" event={"ID":"4e52a62f-f2fa-4b66-be91-758b8dffcfbd","Type":"ContainerDied","Data":"e03a457b9aa72b8f810122d05338d07a4709b298e247a01d6e8f69b5dd640ab6"}
Sep 17 17:37:41.511001 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.510993    2563 scope.go:117] "RemoveContainer" containerID="8621205ac47976c62909995856f2e17fa98b2309bfde0199a45b1cfa5352bb5a"
Sep 17 17:37:41.511223 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.511207    2563 scope.go:117] "RemoveContainer" containerID="e03a457b9aa72b8f810122d05338d07a4709b298e247a01d6e8f69b5dd640ab6"
Sep 17 17:37:41.511395 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:41.511378    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-p9xpb_openshift-console-operator(4e52a62f-f2fa-4b66-be91-758b8dffcfbd)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" podUID="4e52a62f-f2fa-4b66-be91-758b8dffcfbd"
Sep 17 17:37:41.662679 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.662643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.764004 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:41.763913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.434299 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.434265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.514916 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.514883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.515342 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.514988    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:37:42.515386 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.515356    2563 scope.go:117] "RemoveContainer" containerID="e03a457b9aa72b8f810122d05338d07a4709b298e247a01d6e8f69b5dd640ab6"
Sep 17 17:37:42.515541 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:42.515511    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-p9xpb_openshift-console-operator(4e52a62f-f2fa-4b66-be91-758b8dffcfbd)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" podUID="4e52a62f-f2fa-4b66-be91-758b8dffcfbd"
Sep 17 17:37:42.645410 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.645381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.742102 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.742019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.844462 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:42.844429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:43.585710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.585664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:43.585710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.585705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:43.586227 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:43.585820    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 17:37:43.586227 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:43.585844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:59.585826256 +0000 UTC m=+165.113608065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : configmap references non-existent config key: service-ca.crt
Sep 17 17:37:43.586227 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:43.585865    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs podName:6abc45d9-b9e0-4f47-9819-97ad80b332ee nodeName:}" failed. No retries permitted until 2026-09-17 17:37:59.585859075 +0000 UTC m=+165.113640879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs") pod "router-default-56879d8b98-cgl46" (UID: "6abc45d9-b9e0-4f47-9819-97ad80b332ee") : secret "router-metrics-certs-default" not found
Sep 17 17:37:43.964805 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.964768    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-4vcj4"]
Sep 17 17:37:43.967188 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.967167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:43.969617 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.969592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:37:43.969719 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.969637    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:37:43.969719 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.969603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:37:43.970288 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.970273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xvv9w\""
Sep 17 17:37:43.970381 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.970290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:37:43.977288 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.977266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-4vcj4"]
Sep 17 17:37:43.988700 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.988681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-signing-key\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:43.988809 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.988723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-signing-cabundle\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:43.988809 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:43.988768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9kcq\" (UniqueName: \"kubernetes.io/projected/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-kube-api-access-p9kcq\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.008346 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.008318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:44.071331 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.071292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:44.089984 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.089959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-signing-key\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.090116 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.090008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-signing-cabundle\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.090116 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.090035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9kcq\" (UniqueName: \"kubernetes.io/projected/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-kube-api-access-p9kcq\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.090728 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.090701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-signing-cabundle\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.092512 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.092488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-signing-key\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.101105 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.101080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9kcq\" (UniqueName: \"kubernetes.io/projected/c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a-kube-api-access-p9kcq\") pod \"service-ca-55f6fc7b86-4vcj4\" (UID: \"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.275435 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.275334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4"
Sep 17 17:37:44.542315 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.542285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-4vcj4"]
Sep 17 17:37:44.544771 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:37:44.544744    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f9c1b9_710c_4c37_9a38_07e70b2ffa5a.slice/crio-2fbfd9ebec1f3fa612bd706f973a60605cd8bac4b74b705a0271fa1b8b9bf9cd WatchSource:0}: Error finding container 2fbfd9ebec1f3fa612bd706f973a60605cd8bac4b74b705a0271fa1b8b9bf9cd: Status 404 returned error can't find the container with id 2fbfd9ebec1f3fa612bd706f973a60605cd8bac4b74b705a0271fa1b8b9bf9cd
Sep 17 17:37:44.556190 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.556165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:44.632723 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.632689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:44.995387 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:44.995357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:45.523330 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:45.523291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4" event={"ID":"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a","Type":"ContainerStarted","Data":"2fbfd9ebec1f3fa612bd706f973a60605cd8bac4b74b705a0271fa1b8b9bf9cd"}
Sep 17 17:37:45.524810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:45.524770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:46.527228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:46.527132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4" event={"ID":"c5f9c1b9-710c-4c37-9a38-07e70b2ffa5a","Type":"ContainerStarted","Data":"3548f3641c23673ee1158ff91917f54ea34e93c6ae6931067e6b6675d8a1ae18"}
Sep 17 17:37:46.544073 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:46.544003    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-4vcj4" podStartSLOduration=1.836546329 podStartE2EDuration="3.5439805s" totalImagesPullingTime="1.707434171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:46.543063919 +0000 UTC m=+152.070845747" watchObservedRunningTime="2026-09-17 17:37:46.5439805 +0000 UTC m=+152.071762326"
Sep 17 17:37:47.337486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.337455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:47.439544 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.439514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:47.535164 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.535124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:47.634758 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.634672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:47.722187 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.722134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:47.722398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.722202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:47.722514 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:47.722493    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:47.722578 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:47.722572    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls podName:38a82660-d31e-4922-a63e-98d2ccf89243 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:03.722550713 +0000 UTC m=+169.250332519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pxzlk" (UID: "38a82660-d31e-4922-a63e-98d2ccf89243") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:47.723714 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.723693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38a82660-d31e-4922-a63e-98d2ccf89243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:37:47.771712 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.771675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:47.888975 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:47.888878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:48.055529 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:48.055486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:48.055702 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:48.055547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:37:48.055916 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:48.055904    2563 scope.go:117] "RemoveContainer" containerID="e03a457b9aa72b8f810122d05338d07a4709b298e247a01d6e8f69b5dd640ab6"
Sep 17 17:37:48.056064 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:48.056048    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-p9xpb_openshift-console-operator(4e52a62f-f2fa-4b66-be91-758b8dffcfbd)\"" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" podUID="4e52a62f-f2fa-4b66-be91-758b8dffcfbd"
Sep 17 17:37:48.110286 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:48.110252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:48.945884 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:48.945848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:49.278738 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:49.278645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:49.472093 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:49.472050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:49.671921 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:49.671887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:49.873472 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:49.873438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:50.071669 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:50.071575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:51.181116 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.181084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:51.281218 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.281181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:51.319460 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:51.319414    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" podUID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686"
Sep 17 17:37:51.355940 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:51.355907    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zmsks" podUID="9db7af89-b3ff-44d8-8354-525579dcdf5b"
Sep 17 17:37:51.381616 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.381589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:51.485532 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.485446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:51.540527 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.540499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmsks"
Sep 17 17:37:51.540746 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.540499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:37:51.681688 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.681646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:51.878923 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:51.878833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.114337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:52.114310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.256270 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:52.256239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.972120 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:37:52.972079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zxjft" podUID="d2d139ed-fedb-42f4-b2d7-10e810b16090"
Sep 17 17:37:53.072648 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:53.072610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.266225 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:53.266123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.478498 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:53.478465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.674443 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:53.674407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.869779 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:53.869748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:54.471581 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:54.471546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:54.867465 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:54.867362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:55.276604 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:55.276566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:55.522149 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:55.522114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:56.069563 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.069524    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:56.293048 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.293002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:37:56.293048 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.293047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:37:56.295383 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.295352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9db7af89-b3ff-44d8-8354-525579dcdf5b-metrics-tls\") pod \"dns-default-zmsks\" (UID: \"9db7af89-b3ff-44d8-8354-525579dcdf5b\") " pod="openshift-dns/dns-default-zmsks"
Sep 17 17:37:56.295383 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.295376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"image-registry-6b9c5577bd-tm4nb\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") " pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:37:56.344492 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.344406    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksc59\""
Sep 17 17:37:56.344492 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.344406    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l58cx\""
Sep 17 17:37:56.352376 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.352353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmsks"
Sep 17 17:37:56.352516 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.352502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:37:56.424474 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.424353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:56.493917 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.493881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:56.632446 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.632367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmsks"]
Sep 17 17:37:56.635168 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.635146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9c5577bd-tm4nb"]
Sep 17 17:37:56.636193 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:37:56.636113    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db7af89_b3ff_44d8_8354_525579dcdf5b.slice/crio-38fe8a51fb4825d75bdf64818535ee667ede0a8e47dce9e205fc5d8085b832ae WatchSource:0}: Error finding container 38fe8a51fb4825d75bdf64818535ee667ede0a8e47dce9e205fc5d8085b832ae: Status 404 returned error can't find the container with id 38fe8a51fb4825d75bdf64818535ee667ede0a8e47dce9e205fc5d8085b832ae
Sep 17 17:37:56.637997 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:37:56.637908    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb44e32_45d0_4fde_b7cf_0384ab5f1686.slice/crio-6c50b9719e6a93eb9b17fe2579c609931ec1c93861e44f2fbd149c57f3ffb0ec WatchSource:0}: Error finding container 6c50b9719e6a93eb9b17fe2579c609931ec1c93861e44f2fbd149c57f3ffb0ec: Status 404 returned error can't find the container with id 6c50b9719e6a93eb9b17fe2579c609931ec1c93861e44f2fbd149c57f3ffb0ec
Sep 17 17:37:56.861452 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:56.861411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:57.262872 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.262837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:57.558902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.558810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmsks" event={"ID":"9db7af89-b3ff-44d8-8354-525579dcdf5b","Type":"ContainerStarted","Data":"38fe8a51fb4825d75bdf64818535ee667ede0a8e47dce9e205fc5d8085b832ae"}
Sep 17 17:37:57.560444 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.560415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" event={"ID":"3bb44e32-45d0-4fde-b7cf-0384ab5f1686","Type":"ContainerStarted","Data":"9f8f2dd0515b3fcb450a9f0f1e9a74d0aa89eafdaac5cafa10b674c659e3a899"}
Sep 17 17:37:57.560584 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.560468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" event={"ID":"3bb44e32-45d0-4fde-b7cf-0384ab5f1686","Type":"ContainerStarted","Data":"6c50b9719e6a93eb9b17fe2579c609931ec1c93861e44f2fbd149c57f3ffb0ec"}
Sep 17 17:37:57.560584 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.560545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:37:57.583862 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.583781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" podStartSLOduration=150.583759867 podStartE2EDuration="2m30.583759867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:57.583120983 +0000 UTC m=+163.110902820" watchObservedRunningTime="2026-09-17 17:37:57.583759867 +0000 UTC m=+163.111541694"
Sep 17 17:37:57.664472 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.664435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:57.876498 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:57.876402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:58.118994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:58.118961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:58.568305 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:58.568266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmsks" event={"ID":"9db7af89-b3ff-44d8-8354-525579dcdf5b","Type":"ContainerStarted","Data":"b3c076c1d247ab2a929d9c0f55b8b6fe15940954d0e41c67f9b02952b8da427e"}
Sep 17 17:37:58.568477 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:58.568348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmsks" event={"ID":"9db7af89-b3ff-44d8-8354-525579dcdf5b","Type":"ContainerStarted","Data":"ea8ed76d13125c964d9b1fbd239276ba1a4327430861242b1bc77e7c11db615f"}
Sep 17 17:37:58.568552 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:58.568520    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmsks"
Sep 17 17:37:58.587240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:58.587188    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zmsks" podStartSLOduration=129.065355277 podStartE2EDuration="2m10.587171901s" totalImagesPullingTime="1.521816624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:58.586246737 +0000 UTC m=+164.114028576" watchObservedRunningTime="2026-09-17 17:37:58.587171901 +0000 UTC m=+164.114953727"
Sep 17 17:37:59.623877 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:59.623836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:59.624367 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:59.623916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:59.624502 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:59.624483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6abc45d9-b9e0-4f47-9819-97ad80b332ee-service-ca-bundle\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:59.626288 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:59.626268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6abc45d9-b9e0-4f47-9819-97ad80b332ee-metrics-certs\") pod \"router-default-56879d8b98-cgl46\" (UID: \"6abc45d9-b9e0-4f47-9819-97ad80b332ee\") " pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:37:59.818891 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:37:59.818843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:38:00.098244 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.098128    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56879d8b98-cgl46"]
Sep 17 17:38:00.100714 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:00.100675    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abc45d9_b9e0_4f47_9819_97ad80b332ee.slice/crio-29d493d5e11e05c7cfbf8178f716250c1e30f899f1e0dee8c52d1505b5639734 WatchSource:0}: Error finding container 29d493d5e11e05c7cfbf8178f716250c1e30f899f1e0dee8c52d1505b5639734: Status 404 returned error can't find the container with id 29d493d5e11e05c7cfbf8178f716250c1e30f899f1e0dee8c52d1505b5639734
Sep 17 17:38:00.577832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.577775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56879d8b98-cgl46" event={"ID":"6abc45d9-b9e0-4f47-9819-97ad80b332ee","Type":"ContainerStarted","Data":"ffb2c4f953a8d1433afa4e12ff470ccda687f423600ac780a313349479927c8a"}
Sep 17 17:38:00.577984 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.577848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56879d8b98-cgl46" event={"ID":"6abc45d9-b9e0-4f47-9819-97ad80b332ee","Type":"ContainerStarted","Data":"29d493d5e11e05c7cfbf8178f716250c1e30f899f1e0dee8c52d1505b5639734"}
Sep 17 17:38:00.602420 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.602368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56879d8b98-cgl46" podStartSLOduration=33.602346925 podStartE2EDuration="33.602346925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:00.60156867 +0000 UTC m=+166.129350497" watchObservedRunningTime="2026-09-17 17:38:00.602346925 +0000 UTC m=+166.130128752"
Sep 17 17:38:00.819411 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.819369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:38:00.821956 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.821930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:38:00.953480 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:00.953456    2563 scope.go:117] "RemoveContainer" containerID="e03a457b9aa72b8f810122d05338d07a4709b298e247a01d6e8f69b5dd640ab6"
Sep 17 17:38:01.500135 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.500099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:01.573545 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.573506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:01.583662 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.583638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:38:01.583845 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.583759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" event={"ID":"4e52a62f-f2fa-4b66-be91-758b8dffcfbd","Type":"ContainerStarted","Data":"2c64ebf5975c861eb0578f601da997f3d6f96b6d4c1c6af59208a8c7bc9cd3d4"}
Sep 17 17:38:01.584043 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.584028    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:38:01.584120 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.584063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:38:01.585375 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.585356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56879d8b98-cgl46"
Sep 17 17:38:01.589005 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.588986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb"
Sep 17 17:38:01.603277 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.603221    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-p9xpb" podStartSLOduration=22.706603246 podStartE2EDuration="24.603207295s" totalImagesPullingTime="1.896604049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:01.602918342 +0000 UTC m=+167.130700167" watchObservedRunningTime="2026-09-17 17:38:01.603207295 +0000 UTC m=+167.130989121"
Sep 17 17:38:01.643621 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.643590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:01.784746 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.784651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:01.964849 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:01.964814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.061736 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.061646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.162357 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.162315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.317163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.317069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.414468 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.414429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.584447 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.584356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.707028 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.706993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.808723 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:02.808686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:03.172778 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:03.171698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:03.378149 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:03.378115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:03.480810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:03.480682    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:03.761215 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:03.761138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:38:03.763446 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:03.763409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38a82660-d31e-4922-a63e-98d2ccf89243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pxzlk\" (UID: \"38a82660-d31e-4922-a63e-98d2ccf89243\") " pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:38:03.939095 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:03.939063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pxzlk"
Sep 17 17:38:04.236248 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:04.236216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pxzlk"]
Sep 17 17:38:04.240058 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:04.240035    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a82660_d31e_4922_a63e_98d2ccf89243.slice/crio-b68ee35a5acffdeef37c71ad0a1645625a1d1dc96602c7245da4492b2e3b97c2 WatchSource:0}: Error finding container b68ee35a5acffdeef37c71ad0a1645625a1d1dc96602c7245da4492b2e3b97c2: Status 404 returned error can't find the container with id b68ee35a5acffdeef37c71ad0a1645625a1d1dc96602c7245da4492b2e3b97c2
Sep 17 17:38:04.593447 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:04.593364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxzlk" event={"ID":"38a82660-d31e-4922-a63e-98d2ccf89243","Type":"ContainerStarted","Data":"c3ccf916974e1ce8591a8ae069efd60d2b897176607dc1a607b8daf0f9367364"}
Sep 17 17:38:04.593447 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:04.593426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxzlk" event={"ID":"38a82660-d31e-4922-a63e-98d2ccf89243","Type":"ContainerStarted","Data":"b68ee35a5acffdeef37c71ad0a1645625a1d1dc96602c7245da4492b2e3b97c2"}
Sep 17 17:38:04.595741 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:04.595717    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:04.659321 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:04.659283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:05.350350 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.350320    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b9c5577bd-tm4nb"]
Sep 17 17:38:05.409728 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.409681    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"]
Sep 17 17:38:05.411977 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.411949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"
Sep 17 17:38:05.414308 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.414286    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6rthg\""
Sep 17 17:38:05.453450 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.453415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"]
Sep 17 17:38:05.476533 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.476489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9457fa88-f119-457f-82bd-e3aaa296c5f3-kube-api-access-9prmj\") pod \"network-check-source-7cbc45fddc-np47h\" (UID: \"9457fa88-f119-457f-82bd-e3aaa296c5f3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"
Sep 17 17:38:05.510761 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.510725    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-p62zj"]
Sep 17 17:38:05.512846 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.512817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:05.516054 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.516027    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hk6ss\""
Sep 17 17:38:05.516492 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.516303    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:38:05.516608 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.516559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:38:05.577362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.577328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9457fa88-f119-457f-82bd-e3aaa296c5f3-kube-api-access-9prmj\") pod \"network-check-source-7cbc45fddc-np47h\" (UID: \"9457fa88-f119-457f-82bd-e3aaa296c5f3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"
Sep 17 17:38:05.577558 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.577397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqgk\" (UniqueName: \"kubernetes.io/projected/40cb78ac-a56b-49f3-805c-1ab809d43d4c-kube-api-access-clqgk\") pod \"downloads-6594f4c75-p62zj\" (UID: \"40cb78ac-a56b-49f3-805c-1ab809d43d4c\") " pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:05.597426 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.597396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prmj\" (UniqueName: \"kubernetes.io/projected/9457fa88-f119-457f-82bd-e3aaa296c5f3-kube-api-access-9prmj\") pod \"network-check-source-7cbc45fddc-np47h\" (UID: \"9457fa88-f119-457f-82bd-e3aaa296c5f3\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"
Sep 17 17:38:05.600203 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.600168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxzlk" event={"ID":"38a82660-d31e-4922-a63e-98d2ccf89243","Type":"ContainerStarted","Data":"f64c9ff43d812395f573db998bedd4eb56af6a17b74d9e789805cb659e9d0958"}
Sep 17 17:38:05.614356 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.614267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-p62zj"]
Sep 17 17:38:05.653381 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.653318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:05.677977 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.677946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqgk\" (UniqueName: \"kubernetes.io/projected/40cb78ac-a56b-49f3-805c-1ab809d43d4c-kube-api-access-clqgk\") pod \"downloads-6594f4c75-p62zj\" (UID: \"40cb78ac-a56b-49f3-805c-1ab809d43d4c\") " pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:05.690332 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.690299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqgk\" (UniqueName: \"kubernetes.io/projected/40cb78ac-a56b-49f3-805c-1ab809d43d4c-kube-api-access-clqgk\") pod \"downloads-6594f4c75-p62zj\" (UID: \"40cb78ac-a56b-49f3-805c-1ab809d43d4c\") " pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:05.723041 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.723005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"
Sep 17 17:38:05.753645 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.753609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:05.823743 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.823707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:05.953145 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:05.953112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:38:06.010365 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.010329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h"]
Sep 17 17:38:06.459535 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.459451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-p62zj"]
Sep 17 17:38:06.462692 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:06.462665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cb78ac_a56b_49f3_805c_1ab809d43d4c.slice/crio-67f64fd0eaebd1186b66787d8a11592687d60f8c352f9e4f2e203b70b3d3af40 WatchSource:0}: Error finding container 67f64fd0eaebd1186b66787d8a11592687d60f8c352f9e4f2e203b70b3d3af40: Status 404 returned error can't find the container with id 67f64fd0eaebd1186b66787d8a11592687d60f8c352f9e4f2e203b70b3d3af40
Sep 17 17:38:06.605399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.605321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pxzlk" event={"ID":"38a82660-d31e-4922-a63e-98d2ccf89243","Type":"ContainerStarted","Data":"8f986e22e9a2fde37cd3e8608a4e30ce69f6fdfd10ddc04bfaeb95f09c7b7f4f"}
Sep 17 17:38:06.606656 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.606629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-p62zj" event={"ID":"40cb78ac-a56b-49f3-805c-1ab809d43d4c","Type":"ContainerStarted","Data":"67f64fd0eaebd1186b66787d8a11592687d60f8c352f9e4f2e203b70b3d3af40"}
Sep 17 17:38:06.608068 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.608044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h" event={"ID":"9457fa88-f119-457f-82bd-e3aaa296c5f3","Type":"ContainerStarted","Data":"73c63d7f88621c369f3bc83206a398d72cb3994a9465ee8707c59244b4160108"}
Sep 17 17:38:06.608180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.608087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h" event={"ID":"9457fa88-f119-457f-82bd-e3aaa296c5f3","Type":"ContainerStarted","Data":"5b774d732e079fb2f2d67e22fc3606149e01b93ea8455e3e824b3ccbe6095ca7"}
Sep 17 17:38:06.653210 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.653146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pxzlk" podStartSLOduration=33.805471649 podStartE2EDuration="35.653122758s" totalImagesPullingTime="1.847651109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:06.629865395 +0000 UTC m=+172.157647220" watchObservedRunningTime="2026-09-17 17:38:06.653122758 +0000 UTC m=+172.180904578"
Sep 17 17:38:06.653739 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:06.653712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-np47h" podStartSLOduration=1.6536981229999999 podStartE2EDuration="1.653698123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:06.652831201 +0000 UTC m=+172.180613026" watchObservedRunningTime="2026-09-17 17:38:06.653698123 +0000 UTC m=+172.181479950"
Sep 17 17:38:07.904296 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.904217    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68df7cdfd6-6vh5j"]
Sep 17 17:38:07.906194 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.906165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:07.908874 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.908850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:38:07.909163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.909145    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:38:07.909636 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.909612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t4hbt\""
Sep 17 17:38:07.909735 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.909701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:38:07.909913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.909896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:38:07.910353 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.910098    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:38:07.918902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.918877    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68df7cdfd6-6vh5j"]
Sep 17 17:38:07.998090 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.998051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-oauth-config\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:07.998290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.998099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-service-ca\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:07.998290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.998146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-config\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:07.998290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.998204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgh5w\" (UniqueName: \"kubernetes.io/projected/adcc76e6-46f8-4af7-8058-e20b872b35f7-kube-api-access-kgh5w\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:07.998290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.998241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-serving-cert\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:07.998290 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:07.998271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-oauth-serving-cert\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.099504 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.099457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-oauth-serving-cert\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.099669 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.099522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-oauth-config\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.099669 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.099567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-service-ca\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.099669 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.099615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-config\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.099669 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.099653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgh5w\" (UniqueName: \"kubernetes.io/projected/adcc76e6-46f8-4af7-8058-e20b872b35f7-kube-api-access-kgh5w\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.099930 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.099691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-serving-cert\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.100384 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.100356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-oauth-serving-cert\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.100494 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.100378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-config\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.100494 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.100469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-service-ca\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.102718 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.102696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-serving-cert\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.102912 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.102889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-oauth-config\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.108485 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.108455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgh5w\" (UniqueName: \"kubernetes.io/projected/adcc76e6-46f8-4af7-8058-e20b872b35f7-kube-api-access-kgh5w\") pod \"console-68df7cdfd6-6vh5j\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") " pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.218339 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.218298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:08.234646 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.234580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:08.345455 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.345375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:08.426099 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.426061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:08.499455 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.499247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68df7cdfd6-6vh5j"]
Sep 17 17:38:08.502414 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:08.502384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcc76e6_46f8_4af7_8058_e20b872b35f7.slice/crio-b83a6a0aa9859ca612cbdc1456f97a4100500791068f41413be383147e09dfd6 WatchSource:0}: Error finding container b83a6a0aa9859ca612cbdc1456f97a4100500791068f41413be383147e09dfd6: Status 404 returned error can't find the container with id b83a6a0aa9859ca612cbdc1456f97a4100500791068f41413be383147e09dfd6
Sep 17 17:38:08.573906 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.573851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmsks"
Sep 17 17:38:08.619180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.619141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df7cdfd6-6vh5j" event={"ID":"adcc76e6-46f8-4af7-8058-e20b872b35f7","Type":"ContainerStarted","Data":"b83a6a0aa9859ca612cbdc1456f97a4100500791068f41413be383147e09dfd6"}
Sep 17 17:38:08.690536 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:08.690483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:10.401186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:10.401007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.523669 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.523599    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.679525 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.679468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.785237 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.785152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.818677 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.818637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"]
Sep 17 17:38:12.826689 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.826652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.829459 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.829175    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:38:12.829459 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.829258    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qfpld\""
Sep 17 17:38:12.830332 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.830296    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:38:12.830761 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.830582    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:38:12.830923 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.830867    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:38:12.831215 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.831059    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:38:12.831603 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.831432    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:38:12.835477 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.835452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"]
Sep 17 17:38:12.906121 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.906082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:12.938348 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.938313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cz4rq"]
Sep 17 17:38:12.949839 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.949578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.949839 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.949641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17e59f32-2f84-4564-911d-a92a741ce373-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.950168 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.950131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:12.950763 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.950277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17e59f32-2f84-4564-911d-a92a741ce373-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.950763 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.950329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.950763 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.950381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.950763 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.950425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vm8x\" (UniqueName: \"kubernetes.io/projected/17e59f32-2f84-4564-911d-a92a741ce373-kube-api-access-8vm8x\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:12.953944 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.953175    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:38:12.953944 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.953562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:38:12.953944 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.953783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:38:12.955964 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:12.955947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5v6jf\""
Sep 17 17:38:13.051777 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d315ace-900d-42d6-960d-cdac601e70b5-metrics-client-ca\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.051777 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vm8x\" (UniqueName: \"kubernetes.io/projected/17e59f32-2f84-4564-911d-a92a741ce373-kube-api-access-8vm8x\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5pd\" (UniqueName: \"kubernetes.io/projected/5d315ace-900d-42d6-960d-cdac601e70b5-kube-api-access-2p5pd\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-tls\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.051987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-root\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-wtmp\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17e59f32-2f84-4564-911d-a92a741ce373-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17e59f32-2f84-4564-911d-a92a741ce373-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-textfile\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-sys\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:38:13.052427    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.052454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/17e59f32-2f84-4564-911d-a92a741ce373-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.052521 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:38:13.052488    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-tls podName:17e59f32-2f84-4564-911d-a92a741ce373 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:13.552469086 +0000 UTC m=+179.080250904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-rpqc9" (UID: "17e59f32-2f84-4564-911d-a92a741ce373") : secret "kube-state-metrics-tls" not found
Sep 17 17:38:13.053153 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.053090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17e59f32-2f84-4564-911d-a92a741ce373-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.053269 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.053212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.057124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.057074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.062898 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.062868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vm8x\" (UniqueName: \"kubernetes.io/projected/17e59f32-2f84-4564-911d-a92a741ce373-kube-api-access-8vm8x\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.116857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.116822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:13.152937 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.152905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-wtmp\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.152972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-textfile\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-sys\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d315ace-900d-42d6-960d-cdac601e70b5-metrics-client-ca\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153086 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-wtmp\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153298 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153298 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5pd\" (UniqueName: \"kubernetes.io/projected/5d315ace-900d-42d6-960d-cdac601e70b5-kube-api-access-2p5pd\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153298 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-tls\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153298 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153298 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-root\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153541 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-root\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153541 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-textfile\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.153770 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.153734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d315ace-900d-42d6-960d-cdac601e70b5-sys\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.154146 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.154094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.154315 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.154288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d315ace-900d-42d6-960d-cdac601e70b5-metrics-client-ca\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.155706 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.155685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.156204 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.156172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d315ace-900d-42d6-960d-cdac601e70b5-node-exporter-tls\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.163377 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.163351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5pd\" (UniqueName: \"kubernetes.io/projected/5d315ace-900d-42d6-960d-cdac601e70b5-kube-api-access-2p5pd\") pod \"node-exporter-cz4rq\" (UID: \"5d315ace-900d-42d6-960d-cdac601e70b5\") " pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.217336 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.217229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:13.264058 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.264022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cz4rq"
Sep 17 17:38:13.274991 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:13.274951    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d315ace_900d_42d6_960d_cdac601e70b5.slice/crio-c88f324ef31aa6deaafdc1cd6b61587d56c6d245ba258bc414c1aee17aab524f WatchSource:0}: Error finding container c88f324ef31aa6deaafdc1cd6b61587d56c6d245ba258bc414c1aee17aab524f: Status 404 returned error can't find the container with id c88f324ef31aa6deaafdc1cd6b61587d56c6d245ba258bc414c1aee17aab524f
Sep 17 17:38:13.557179 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.557139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.560038 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.560001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e59f32-2f84-4564-911d-a92a741ce373-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-rpqc9\" (UID: \"17e59f32-2f84-4564-911d-a92a741ce373\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.643133 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.643086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz4rq" event={"ID":"5d315ace-900d-42d6-960d-cdac601e70b5","Type":"ContainerStarted","Data":"c88f324ef31aa6deaafdc1cd6b61587d56c6d245ba258bc414c1aee17aab524f"}
Sep 17 17:38:13.644943 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.644895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df7cdfd6-6vh5j" event={"ID":"adcc76e6-46f8-4af7-8058-e20b872b35f7","Type":"ContainerStarted","Data":"a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe"}
Sep 17 17:38:13.667575 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.667508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68df7cdfd6-6vh5j" podStartSLOduration=2.047266637 podStartE2EDuration="6.667484989s" totalImagesPullingTime="4.620218352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:13.665563959 +0000 UTC m=+179.193345785" watchObservedRunningTime="2026-09-17 17:38:13.667484989 +0000 UTC m=+179.195266815"
Sep 17 17:38:13.742664 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.742458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"
Sep 17 17:38:13.780042 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.779977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:13.869808 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.869616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:13.878045 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.875874    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:13.880021 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.879997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.882430 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.882405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:38:13.882702 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.882686    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:38:13.883295 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.882950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qr2b\""
Sep 17 17:38:13.883295 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883072    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:38:13.883295 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883101    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:38:13.883295 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:38:13.883523 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883433    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:38:13.883869 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:38:13.883869 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883712    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:38:13.883869 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.883734    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:38:13.900380 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.900281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:13.961811 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.961764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.961994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.961841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-web-config\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.961994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.961929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962102 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tlvk\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-kube-api-access-2tlvk\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962102 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-out\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962102 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962252 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962252 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962252 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962252 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962452 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962452 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.962452 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.962366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:13.986811 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:13.986762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:14.034482 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.034395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-rpqc9"]
Sep 17 17:38:14.038825 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:14.038775    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e59f32_2f84_4564_911d_a92a741ce373.slice/crio-c40f819f23fe750176087d2ce2ac7bbe32bec200c169f7acf3a41b41c40e1869 WatchSource:0}: Error finding container c40f819f23fe750176087d2ce2ac7bbe32bec200c169f7acf3a41b41c40e1869: Status 404 returned error can't find the container with id c40f819f23fe750176087d2ce2ac7bbe32bec200c169f7acf3a41b41c40e1869
Sep 17 17:38:14.063126 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063126 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-web-config\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tlvk\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-kube-api-access-2tlvk\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-out\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063706 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063706 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063706 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.063706 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.063512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.065442 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.064276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.065442 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.064825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.065442 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:38:14.064952    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle podName:0e94cce6-f40c-47df-9f51-2c5013e8b641 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:14.56493172 +0000 UTC m=+180.092713531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641") : configmap references non-existent config key: ca-bundle.crt
Sep 17 17:38:14.065442 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:38:14.064971    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 17:38:14.065442 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:38:14.065040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls podName:0e94cce6-f40c-47df-9f51-2c5013e8b641 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:14.565022587 +0000 UTC m=+180.092804404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641") : secret "alertmanager-main-tls" not found
Sep 17 17:38:14.066398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.066330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.066768 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.066725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.070071 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.070024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.070743 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.070698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-out\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.072111 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.072059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.072323 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.072299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-web-config\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.072393 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.072337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.073868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.073815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tlvk\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-kube-api-access-2tlvk\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.074723 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.074702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.156961 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.156868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:14.388853 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.388742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:14.490240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.490207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:14.567675 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.567497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.567675 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.567616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.568626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.568599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.570893 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.570826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:14.652189 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.652045    2563 generic.go:356] "Generic (PLEG): container finished" podID="5d315ace-900d-42d6-960d-cdac601e70b5" containerID="c15a39d07545efb9d3c28ceed81997b05fa4a92cb8c64c35b879531b009cb1de" exitCode=0
Sep 17 17:38:14.652189 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.652134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz4rq" event={"ID":"5d315ace-900d-42d6-960d-cdac601e70b5","Type":"ContainerDied","Data":"c15a39d07545efb9d3c28ceed81997b05fa4a92cb8c64c35b879531b009cb1de"}
Sep 17 17:38:14.653659 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.653633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9" event={"ID":"17e59f32-2f84-4564-911d-a92a741ce373","Type":"ContainerStarted","Data":"c40f819f23fe750176087d2ce2ac7bbe32bec200c169f7acf3a41b41c40e1869"}
Sep 17 17:38:14.798501 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.798296    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qr2b\""
Sep 17 17:38:14.806199 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:14.805924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:38:15.106867 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.106833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:38:15.278694 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:15.278653    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e94cce6_f40c_47df_9f51_2c5013e8b641.slice/crio-f80158b580cde4a09d0c9c995c483a01c6a83a8f8193eeb510fdaff2c7a91e52 WatchSource:0}: Error finding container f80158b580cde4a09d0c9c995c483a01c6a83a8f8193eeb510fdaff2c7a91e52: Status 404 returned error can't find the container with id f80158b580cde4a09d0c9c995c483a01c6a83a8f8193eeb510fdaff2c7a91e52
Sep 17 17:38:15.357182 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.357122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:38:15.661020 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.660934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz4rq" event={"ID":"5d315ace-900d-42d6-960d-cdac601e70b5","Type":"ContainerStarted","Data":"d4cb664c987ca1ef9c0645a6055dddd457782bedd596e1153d798b66714232ed"}
Sep 17 17:38:15.661020 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.661002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz4rq" event={"ID":"5d315ace-900d-42d6-960d-cdac601e70b5","Type":"ContainerStarted","Data":"27861972d2a54277cb697615aa68b9090474f760c4018897bd51143fc4b19b0b"}
Sep 17 17:38:15.663686 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.663642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9" event={"ID":"17e59f32-2f84-4564-911d-a92a741ce373","Type":"ContainerStarted","Data":"a140f0d917f21b06605b1a8c2bcc72fde5d35ecdcff60cac7abe5c91405664c3"}
Sep 17 17:38:15.663840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.663709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9" event={"ID":"17e59f32-2f84-4564-911d-a92a741ce373","Type":"ContainerStarted","Data":"ba1c958d1e6af698cf7de981a7d3be40af3f19e83703c7798eeecd4cf79b5fcb"}
Sep 17 17:38:15.663840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.663740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9" event={"ID":"17e59f32-2f84-4564-911d-a92a741ce373","Type":"ContainerStarted","Data":"52a65eab50d5027047c5a799fd5c1c87f8871662256023ab1b9577f338a89172"}
Sep 17 17:38:15.665178 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.665140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"f80158b580cde4a09d0c9c995c483a01c6a83a8f8193eeb510fdaff2c7a91e52"}
Sep 17 17:38:15.688337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.688280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cz4rq" podStartSLOduration=2.552456625 podStartE2EDuration="3.688257375s" totalImagesPullingTime="1.055327956s" totalInitContainerRuntime="80.472794ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:15.686564503 +0000 UTC m=+181.214346330" watchObservedRunningTime="2026-09-17 17:38:15.688257375 +0000 UTC m=+181.216039204"
Sep 17 17:38:15.707108 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.705396    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-rpqc9" podStartSLOduration=2.438442719 podStartE2EDuration="3.705373727s" totalImagesPullingTime="1.266931008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:15.705033251 +0000 UTC m=+181.232815078" watchObservedRunningTime="2026-09-17 17:38:15.705373727 +0000 UTC m=+181.233155554"
Sep 17 17:38:15.720832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.720783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:15.785735 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.785555    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c88cf684-q7w5t"]
Sep 17 17:38:15.790095 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.790072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.795860 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.795660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:38:15.795860 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.795707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:38:15.795860 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.795740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:38:15.796108 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.795945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:38:15.796186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.796143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:38:15.800385 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.796333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8rci4ms85j7ve\""
Sep 17 17:38:15.800385 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.796378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sfqzq\""
Sep 17 17:38:15.807357 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.806000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c88cf684-q7w5t"]
Sep 17 17:38:15.810904 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.810809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:15.883750 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.883628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.883750 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.883671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.884030 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.883775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-tls\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.884030 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.883883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8m7\" (UniqueName: \"kubernetes.io/projected/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-kube-api-access-pq8m7\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.884030 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.883933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-grpc-tls\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.884030 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.883980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.884030 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.884027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.884267 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.884054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-metrics-client-ca\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.985607 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.985563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8m7\" (UniqueName: \"kubernetes.io/projected/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-kube-api-access-pq8m7\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.985853 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.985619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-grpc-tls\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.985853 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.985656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.986054 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.986027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.986126 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.986073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-metrics-client-ca\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.986180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.986158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.986237 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.986191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.986288 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.986247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-tls\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.987244 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.987216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-metrics-client-ca\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.990229 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.990106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.990810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.990497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.990810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.990754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-grpc-tls\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.991973 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.991946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.992557 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.992533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-tls\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.993088 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.993066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:15.995738 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:15.995716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8m7\" (UniqueName: \"kubernetes.io/projected/6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23-kube-api-access-pq8m7\") pod \"thanos-querier-5c88cf684-q7w5t\" (UID: \"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23\") " pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:16.071010 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.070972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:16.123993 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.123870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:16.353456 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.353371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:16.472969 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.472909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c88cf684-q7w5t"]
Sep 17 17:38:16.474481 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:16.474448    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7ad3e3_d2a8_47f5_97e1_2d3858bb6d23.slice/crio-987a30f19409e518b9056f2bd3aadfbe67af07a38327f94e9921085c8f19771e WatchSource:0}: Error finding container 987a30f19409e518b9056f2bd3aadfbe67af07a38327f94e9921085c8f19771e: Status 404 returned error can't find the container with id 987a30f19409e518b9056f2bd3aadfbe67af07a38327f94e9921085c8f19771e
Sep 17 17:38:16.671536 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.671445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"987a30f19409e518b9056f2bd3aadfbe67af07a38327f94e9921085c8f19771e"}
Sep 17 17:38:16.673409 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.673381    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29" exitCode=0
Sep 17 17:38:16.673547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.673475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29"}
Sep 17 17:38:16.789277 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.789235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:16.860833 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:16.860754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:17.394146 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.393889    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54c789cfc6-cmwjl"]
Sep 17 17:38:17.397277 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.397245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.405013 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.404948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:38:17.407586 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.407562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c789cfc6-cmwjl"]
Sep 17 17:38:17.503705 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.503665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-trusted-ca-bundle\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.503900 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.503772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-config\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.503900 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.503839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2ft\" (UniqueName: \"kubernetes.io/projected/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-kube-api-access-4g2ft\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.504037 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.503919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-oauth-config\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.504037 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.504003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-service-ca\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.504130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.504039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-oauth-serving-cert\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.504130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.504095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-serving-cert\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.543683 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.543639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-config\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2ft\" (UniqueName: \"kubernetes.io/projected/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-kube-api-access-4g2ft\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-oauth-config\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-service-ca\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-oauth-serving-cert\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-serving-cert\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606253 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.605577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-trusted-ca-bundle\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.606751 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.606587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-trusted-ca-bundle\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.607032 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.607002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-oauth-serving-cert\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.607115 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.607061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-config\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.607470 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.607433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-service-ca\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.610023 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.609960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-oauth-config\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.614026 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.613981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-serving-cert\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.616766 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.616720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2ft\" (UniqueName: \"kubernetes.io/projected/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-kube-api-access-4g2ft\") pod \"console-54c789cfc6-cmwjl\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") " pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.646775 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.646690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:17.710637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.710605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:17.985920 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:17.985880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:18.219316 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:18.219256    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:18.219522 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:18.219338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:18.224825 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:18.224782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:18.688149 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:18.688119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:38:18.890161 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:18.890125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:18.992244 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:18.992161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:19.085324 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:19.085293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:23.390909 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.390871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:23.492965 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.492929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:23.634120 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.634084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:23.634733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.634711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c789cfc6-cmwjl"]
Sep 17 17:38:23.677998 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.677957    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b8dbf9ff-zqcfp"]
Sep 17 17:38:23.680945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.680921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.702660 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.702624    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b8dbf9ff-zqcfp"]
Sep 17 17:38:23.733355 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.733313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:23.763589 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-oauth-config\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.763589 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-config\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.763858 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-service-ca\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.763858 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-trusted-ca-bundle\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.763858 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-oauth-serving-cert\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.763990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmd9\" (UniqueName: \"kubernetes.io/projected/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-kube-api-access-2hmd9\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.764028 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.763994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-serving-cert\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865152 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-oauth-config\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865349 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-config\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865349 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-service-ca\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865349 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-trusted-ca-bundle\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865349 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-oauth-serving-cert\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865349 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmd9\" (UniqueName: \"kubernetes.io/projected/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-kube-api-access-2hmd9\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.865597 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.865368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-serving-cert\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.866053 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.866028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-config\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.866377 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.866311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-oauth-serving-cert\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.866377 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.866324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-service-ca\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.866594 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.866454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-trusted-ca-bundle\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.868102 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.868079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-oauth-config\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.868207 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.868141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-serving-cert\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.874476 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.874453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmd9\" (UniqueName: \"kubernetes.io/projected/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-kube-api-access-2hmd9\") pod \"console-58b8dbf9ff-zqcfp\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") " pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:23.991751 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:23.991657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:25.411180 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.411140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:25.520887 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.520856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:25.718467 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.718438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5"}
Sep 17 17:38:25.718590 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.718549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c"}
Sep 17 17:38:25.721037 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.720997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-p62zj" event={"ID":"40cb78ac-a56b-49f3-805c-1ab809d43d4c","Type":"ContainerStarted","Data":"f5c40b166fd12d15531854a472970131b1b699cb0e8717d43662e346dcc13289"}
Sep 17 17:38:25.721542 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.721475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:25.723524 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.723496    2563 patch_prober.go:28] interesting pod/downloads-6594f4c75-p62zj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body=
Sep 17 17:38:25.723616 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.723546    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-p62zj" podUID="40cb78ac-a56b-49f3-805c-1ab809d43d4c" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused"
Sep 17 17:38:25.724841 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.724780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"88cfe99089f959efa57349acf434bfcac821de65321257666558423247fe1d93"}
Sep 17 17:38:25.724986 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.724973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"2439aa95e26d1aaad96edbcb30b7064a582df2c6d2496398a0f30177c1e9cea4"}
Sep 17 17:38:25.743543 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.743494    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-p62zj" podStartSLOduration=1.6583749669999999 podStartE2EDuration="20.743474237s" totalImagesPullingTime="19.08509927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:25.741471808 +0000 UTC m=+191.269253634" watchObservedRunningTime="2026-09-17 17:38:25.743474237 +0000 UTC m=+191.271256064"
Sep 17 17:38:25.785726 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.785698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b8dbf9ff-zqcfp"]
Sep 17 17:38:25.789545 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:38:25.789463    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b78e4f_fa8f_4acd_8451_aa659cf62dbd.slice/crio-d22a415b205caeab2e3328b08296e34665ef55b87936679a725390ba7d19f26a WatchSource:0}: Error finding container d22a415b205caeab2e3328b08296e34665ef55b87936679a725390ba7d19f26a: Status 404 returned error can't find the container with id d22a415b205caeab2e3328b08296e34665ef55b87936679a725390ba7d19f26a
Sep 17 17:38:25.790926 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.790694    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c789cfc6-cmwjl"]
Sep 17 17:38:25.824500 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.824450    2563 patch_prober.go:28] interesting pod/downloads-6594f4c75-p62zj container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body=
Sep 17 17:38:25.824636 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:25.824505    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6594f4c75-p62zj" podUID="40cb78ac-a56b-49f3-805c-1ab809d43d4c" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused"
Sep 17 17:38:26.206828 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.206766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:26.276519 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.276374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:26.737206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.737172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff"}
Sep 17 17:38:26.737710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.737258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff"}
Sep 17 17:38:26.737710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.737302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae"}
Sep 17 17:38:26.737710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.737343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerStarted","Data":"0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b"}
Sep 17 17:38:26.740965 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.740937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"2b52400eb35434bf691238e87a2290c9cb8c49c35bd960deef9e617ec570a07a"}
Sep 17 17:38:26.741078 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.741016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"f48ddb6d720e58abf5df79375daee42a967e2cbbc7d39f03101448642f2f3f7d"}
Sep 17 17:38:26.745969 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.745938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b8dbf9ff-zqcfp" event={"ID":"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd","Type":"ContainerStarted","Data":"52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d"}
Sep 17 17:38:26.746200 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.746181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b8dbf9ff-zqcfp" event={"ID":"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd","Type":"ContainerStarted","Data":"d22a415b205caeab2e3328b08296e34665ef55b87936679a725390ba7d19f26a"}
Sep 17 17:38:26.748861 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.748769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c789cfc6-cmwjl" event={"ID":"09c5cf63-5bbf-41f8-96d6-7860724fe3e3","Type":"ContainerStarted","Data":"a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b"}
Sep 17 17:38:26.748861 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.748830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c789cfc6-cmwjl" event={"ID":"09c5cf63-5bbf-41f8-96d6-7860724fe3e3","Type":"ContainerStarted","Data":"ea5e6ef605f3ddad331fc2d48e31888dc7e09068160de43c32a64e5ce01bedce"}
Sep 17 17:38:26.752914 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.752856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-p62zj"
Sep 17 17:38:26.770203 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.769901    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.970790249 podStartE2EDuration="13.769878661s" totalImagesPullingTime="10.782059795s" totalInitContainerRuntime="17.028617ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:26.769751424 +0000 UTC m=+192.297533254" watchObservedRunningTime="2026-09-17 17:38:26.769878661 +0000 UTC m=+192.297660488"
Sep 17 17:38:26.797205 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.790765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58b8dbf9ff-zqcfp" podStartSLOduration=3.790742398 podStartE2EDuration="3.790742398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:26.788476151 +0000 UTC m=+192.316257978" watchObservedRunningTime="2026-09-17 17:38:26.790742398 +0000 UTC m=+192.318524226"
Sep 17 17:38:26.811026 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:26.810615    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54c789cfc6-cmwjl" podStartSLOduration=9.810593284 podStartE2EDuration="9.810593284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:26.808264284 +0000 UTC m=+192.336046117" watchObservedRunningTime="2026-09-17 17:38:26.810593284 +0000 UTC m=+192.338375112"
Sep 17 17:38:27.422093 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.422054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:27.522515 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.522478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:27.705771 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.705680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:27.711458 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.711428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:27.757368 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.757326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"118e6b2e149b4f077d41bcfd8deff445118b395706b06e44aa8e166b5a501f08"}
Sep 17 17:38:27.757913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.757405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" event={"ID":"6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23","Type":"ContainerStarted","Data":"0e54582603e2ccc7050a7dfa1225db6c929a0028185fa1743502ee16272b7e62"}
Sep 17 17:38:27.785196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.785130    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t" podStartSLOduration=2.897125622 podStartE2EDuration="12.785109274s" totalImagesPullingTime="9.887983652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:27.783711765 +0000 UTC m=+193.311493608" watchObservedRunningTime="2026-09-17 17:38:27.785109274 +0000 UTC m=+193.312891101"
Sep 17 17:38:27.789646 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:27.789609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:28.035578 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:28.035495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:28.120096 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:28.120057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:28.762888 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:28.762854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:30.374733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:30.374690    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" podUID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686" containerName="registry" containerID="cri-o://9f8f2dd0515b3fcb450a9f0f1e9a74d0aa89eafdaac5cafa10b674c659e3a899" gracePeriod=30
Sep 17 17:38:30.773640 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:30.773605    2563 generic.go:356] "Generic (PLEG): container finished" podID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686" containerID="9f8f2dd0515b3fcb450a9f0f1e9a74d0aa89eafdaac5cafa10b674c659e3a899" exitCode=0
Sep 17 17:38:30.773825 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:30.773656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" event={"ID":"3bb44e32-45d0-4fde-b7cf-0384ab5f1686","Type":"ContainerDied","Data":"9f8f2dd0515b3fcb450a9f0f1e9a74d0aa89eafdaac5cafa10b674c659e3a899"}
Sep 17 17:38:31.155889 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.155761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:31.192752 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.192716    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:38:31.236380 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.235681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:31.239976 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.239951    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-certificates\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.239988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-ca-trust-extracted\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-bound-sa-token\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-trusted-ca\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gccs7\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-kube-api-access-gccs7\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-installation-pull-secrets\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.240398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-image-registry-private-configuration\") pod \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\" (UID: \"3bb44e32-45d0-4fde-b7cf-0384ab5f1686\") "
Sep 17 17:38:31.241160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.240683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:31.241883 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.241833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:31.244105 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.244072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:31.248234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.248187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-kube-api-access-gccs7" (OuterVolumeSpecName: "kube-api-access-gccs7") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "kube-api-access-gccs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:31.250613 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.250585    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:31.251414 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.251341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:31.251647 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.251576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:31.253505 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.253447    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3bb44e32-45d0-4fde-b7cf-0384ab5f1686" (UID: "3bb44e32-45d0-4fde-b7cf-0384ab5f1686"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:31.342031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.341939    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-certificates\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.341978    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-registry-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.341992    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-ca-trust-extracted\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.342004    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-bound-sa-token\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.342016    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-trusted-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342031 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.342026    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccs7\" (UniqueName: \"kubernetes.io/projected/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-kube-api-access-gccs7\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342328 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.342038    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-installation-pull-secrets\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.342328 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.342052    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bb44e32-45d0-4fde-b7cf-0384ab5f1686-image-registry-private-configuration\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:31.780764 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.780715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb" event={"ID":"3bb44e32-45d0-4fde-b7cf-0384ab5f1686","Type":"ContainerDied","Data":"6c50b9719e6a93eb9b17fe2579c609931ec1c93861e44f2fbd149c57f3ffb0ec"}
Sep 17 17:38:31.781259 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.780731    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c5577bd-tm4nb"
Sep 17 17:38:31.781259 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.780845    2563 scope.go:117] "RemoveContainer" containerID="9f8f2dd0515b3fcb450a9f0f1e9a74d0aa89eafdaac5cafa10b674c659e3a899"
Sep 17 17:38:31.809162 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.809125    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b9c5577bd-tm4nb"]
Sep 17 17:38:31.816342 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:31.816309    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b9c5577bd-tm4nb"]
Sep 17 17:38:32.337434 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:32.337389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:32.440619 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:32.440583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:32.958749 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:32.958713    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686" path="/var/lib/kubelet/pods/3bb44e32-45d0-4fde-b7cf-0384ab5f1686/volumes"
Sep 17 17:38:33.992648 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:33.992602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:33.993149 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:33.992683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:33.998403 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:33.998374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:34.775127 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:34.775100    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c88cf684-q7w5t"
Sep 17 17:38:34.800008 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:34.799978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:38:34.854602 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:34.854565    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68df7cdfd6-6vh5j"]
Sep 17 17:38:34.969353 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:34.969313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:35.070712 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:35.070601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:35.169758 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:35.169719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:37.662755 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:37.662724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56879d8b98-cgl46_6abc45d9-b9e0-4f47-9819-97ad80b332ee/router/0.log"
Sep 17 17:38:37.674285 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:37.674249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jczs6_4622bbec-ed20-4923-a475-1e138ed7b050/serve-healthcheck-canary/0.log"
Sep 17 17:38:46.128745 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:46.128707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:46.230754 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:46.230718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:46.330546 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:46.330508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:47.333134 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:47.333101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:47.398337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:47.398304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:51.782425 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:51.782361    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54c789cfc6-cmwjl" podUID="09c5cf63-5bbf-41f8-96d6-7860724fe3e3" containerName="console" containerID="cri-o://a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b" gracePeriod=15
Sep 17 17:38:52.065847 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.065742    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:52.133723 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-serving-cert\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.133723 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-trusted-ca-bundle\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.134014 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-config\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.134014 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-service-ca\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.134014 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-oauth-serving-cert\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.134014 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-oauth-config\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.134014 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.133896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g2ft\" (UniqueName: \"kubernetes.io/projected/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-kube-api-access-4g2ft\") pod \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\" (UID: \"09c5cf63-5bbf-41f8-96d6-7860724fe3e3\") "
Sep 17 17:38:52.134261 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.134226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-service-ca" (OuterVolumeSpecName: "service-ca") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:52.134261 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.134239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:52.134362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.134254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-config" (OuterVolumeSpecName: "console-config") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:52.134498 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.134474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:52.136144 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.136117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:52.136259 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.136179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:52.136259 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.136186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-kube-api-access-4g2ft" (OuterVolumeSpecName: "kube-api-access-4g2ft") pod "09c5cf63-5bbf-41f8-96d6-7860724fe3e3" (UID: "09c5cf63-5bbf-41f8-96d6-7860724fe3e3"). InnerVolumeSpecName "kube-api-access-4g2ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:52.235511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235470    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.235511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235505    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-service-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.235511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235517    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-oauth-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.235748 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235525    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-oauth-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.235748 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235534    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2ft\" (UniqueName: \"kubernetes.io/projected/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-kube-api-access-4g2ft\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.235748 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235543    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-console-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.235748 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.235550    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c5cf63-5bbf-41f8-96d6-7860724fe3e3-trusted-ca-bundle\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:38:52.873683 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.873652    2563 generic.go:356] "Generic (PLEG): container finished" podID="09c5cf63-5bbf-41f8-96d6-7860724fe3e3" containerID="a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b" exitCode=0
Sep 17 17:38:52.874158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.873693    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c789cfc6-cmwjl"
Sep 17 17:38:52.874158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.873744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c789cfc6-cmwjl" event={"ID":"09c5cf63-5bbf-41f8-96d6-7860724fe3e3","Type":"ContainerDied","Data":"a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b"}
Sep 17 17:38:52.874158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.873854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c789cfc6-cmwjl" event={"ID":"09c5cf63-5bbf-41f8-96d6-7860724fe3e3","Type":"ContainerDied","Data":"ea5e6ef605f3ddad331fc2d48e31888dc7e09068160de43c32a64e5ce01bedce"}
Sep 17 17:38:52.874158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.873888    2563 scope.go:117] "RemoveContainer" containerID="a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b"
Sep 17 17:38:52.882801 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.882772    2563 scope.go:117] "RemoveContainer" containerID="a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b"
Sep 17 17:38:52.883069 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:38:52.883051    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b\": container with ID starting with a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b not found: ID does not exist" containerID="a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b"
Sep 17 17:38:52.883119 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.883079    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b"} err="failed to get container status \"a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b\": rpc error: code = NotFound desc = could not find container \"a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b\": container with ID starting with a08519e9bb62cd44533abb17781daeee819eac1cc15f07c862e6c09cdd3bf75b not found: ID does not exist"
Sep 17 17:38:52.905429 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.905406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c789cfc6-cmwjl"]
Sep 17 17:38:52.909129 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.909105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54c789cfc6-cmwjl"]
Sep 17 17:38:52.957125 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:52.957095    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c5cf63-5bbf-41f8-96d6-7860724fe3e3" path="/var/lib/kubelet/pods/09c5cf63-5bbf-41f8-96d6-7860724fe3e3/volumes"
Sep 17 17:38:59.880876 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:38:59.880831    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68df7cdfd6-6vh5j" podUID="adcc76e6-46f8-4af7-8058-e20b872b35f7" containerName="console" containerID="cri-o://a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe" gracePeriod=15
Sep 17 17:39:00.156020 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.155927    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:39:00.309374 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-config\") pod \"adcc76e6-46f8-4af7-8058-e20b872b35f7\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") "
Sep 17 17:39:00.309374 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-service-ca\") pod \"adcc76e6-46f8-4af7-8058-e20b872b35f7\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") "
Sep 17 17:39:00.309626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgh5w\" (UniqueName: \"kubernetes.io/projected/adcc76e6-46f8-4af7-8058-e20b872b35f7-kube-api-access-kgh5w\") pod \"adcc76e6-46f8-4af7-8058-e20b872b35f7\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") "
Sep 17 17:39:00.309626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-serving-cert\") pod \"adcc76e6-46f8-4af7-8058-e20b872b35f7\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") "
Sep 17 17:39:00.309728 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-oauth-config\") pod \"adcc76e6-46f8-4af7-8058-e20b872b35f7\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") "
Sep 17 17:39:00.309728 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-oauth-serving-cert\") pod \"adcc76e6-46f8-4af7-8058-e20b872b35f7\" (UID: \"adcc76e6-46f8-4af7-8058-e20b872b35f7\") "
Sep 17 17:39:00.309877 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-config" (OuterVolumeSpecName: "console-config") pod "adcc76e6-46f8-4af7-8058-e20b872b35f7" (UID: "adcc76e6-46f8-4af7-8058-e20b872b35f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:00.309877 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.309844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "adcc76e6-46f8-4af7-8058-e20b872b35f7" (UID: "adcc76e6-46f8-4af7-8058-e20b872b35f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:00.310074 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.310043    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:00.310163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.310078    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-service-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:00.310163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.310130    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "adcc76e6-46f8-4af7-8058-e20b872b35f7" (UID: "adcc76e6-46f8-4af7-8058-e20b872b35f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:00.311687 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.311651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcc76e6-46f8-4af7-8058-e20b872b35f7-kube-api-access-kgh5w" (OuterVolumeSpecName: "kube-api-access-kgh5w") pod "adcc76e6-46f8-4af7-8058-e20b872b35f7" (UID: "adcc76e6-46f8-4af7-8058-e20b872b35f7"). InnerVolumeSpecName "kube-api-access-kgh5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:00.311687 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.311674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "adcc76e6-46f8-4af7-8058-e20b872b35f7" (UID: "adcc76e6-46f8-4af7-8058-e20b872b35f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:00.311829 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.311696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "adcc76e6-46f8-4af7-8058-e20b872b35f7" (UID: "adcc76e6-46f8-4af7-8058-e20b872b35f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:00.411399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.411309    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgh5w\" (UniqueName: \"kubernetes.io/projected/adcc76e6-46f8-4af7-8058-e20b872b35f7-kube-api-access-kgh5w\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:00.411399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.411343    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:00.411399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.411353    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adcc76e6-46f8-4af7-8058-e20b872b35f7-console-oauth-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:00.411399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.411362    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adcc76e6-46f8-4af7-8058-e20b872b35f7-oauth-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:00.910138 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.910104    2563 generic.go:356] "Generic (PLEG): container finished" podID="adcc76e6-46f8-4af7-8058-e20b872b35f7" containerID="a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe" exitCode=0
Sep 17 17:39:00.910590 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.910137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df7cdfd6-6vh5j" event={"ID":"adcc76e6-46f8-4af7-8058-e20b872b35f7","Type":"ContainerDied","Data":"a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe"}
Sep 17 17:39:00.910590 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.910160    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df7cdfd6-6vh5j"
Sep 17 17:39:00.910590 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.910215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df7cdfd6-6vh5j" event={"ID":"adcc76e6-46f8-4af7-8058-e20b872b35f7","Type":"ContainerDied","Data":"b83a6a0aa9859ca612cbdc1456f97a4100500791068f41413be383147e09dfd6"}
Sep 17 17:39:00.910590 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.910236    2563 scope.go:117] "RemoveContainer" containerID="a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe"
Sep 17 17:39:00.919324 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.919307    2563 scope.go:117] "RemoveContainer" containerID="a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe"
Sep 17 17:39:00.919621 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:00.919602    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe\": container with ID starting with a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe not found: ID does not exist" containerID="a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe"
Sep 17 17:39:00.919654 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.919631    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe"} err="failed to get container status \"a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe\": rpc error: code = NotFound desc = could not find container \"a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe\": container with ID starting with a9be72575786216fcdc1eb09f867d8003c6d84c5bcc764192db1114ad49a9abe not found: ID does not exist"
Sep 17 17:39:00.932614 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.932584    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68df7cdfd6-6vh5j"]
Sep 17 17:39:00.937489 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.937467    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68df7cdfd6-6vh5j"]
Sep 17 17:39:00.957234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:00.957205    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adcc76e6-46f8-4af7-8058-e20b872b35f7" path="/var/lib/kubelet/pods/adcc76e6-46f8-4af7-8058-e20b872b35f7/volumes"
Sep 17 17:39:02.322364 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:02.322325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:02.380413 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:02.380384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.398765 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:23.398723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.459037 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:23.459006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:26.640028 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:26.639987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:39:26.642304 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:26.642282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2d139ed-fedb-42f4-b2d7-10e810b16090-metrics-certs\") pod \"network-metrics-daemon-zxjft\" (UID: \"d2d139ed-fedb-42f4-b2d7-10e810b16090\") " pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:39:26.656733 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:26.656706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wr2c\""
Sep 17 17:39:26.664571 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:26.664548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxjft"
Sep 17 17:39:26.937825 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:26.937721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zxjft"]
Sep 17 17:39:26.940704 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:39:26.940676    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d139ed_fedb_42f4_b2d7_10e810b16090.slice/crio-d15a74e12b4aa44611ff182bc8144589ce015bba4ec27b66f62fb2f288df7660 WatchSource:0}: Error finding container d15a74e12b4aa44611ff182bc8144589ce015bba4ec27b66f62fb2f288df7660: Status 404 returned error can't find the container with id d15a74e12b4aa44611ff182bc8144589ce015bba4ec27b66f62fb2f288df7660
Sep 17 17:39:27.012280 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:27.012244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxjft" event={"ID":"d2d139ed-fedb-42f4-b2d7-10e810b16090","Type":"ContainerStarted","Data":"d15a74e12b4aa44611ff182bc8144589ce015bba4ec27b66f62fb2f288df7660"}
Sep 17 17:39:28.018464 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:28.018382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxjft" event={"ID":"d2d139ed-fedb-42f4-b2d7-10e810b16090","Type":"ContainerStarted","Data":"ef2b2fc025cc6e56e2c94ed55e6e76f9fb31df3c721ba3a18c123712b0d9b5be"}
Sep 17 17:39:28.018464 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:28.018441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxjft" event={"ID":"d2d139ed-fedb-42f4-b2d7-10e810b16090","Type":"ContainerStarted","Data":"e07a55310a3dfaf6704ababb726a7fa028014b0271abee4e5855f71d76aa5e0a"}
Sep 17 17:39:28.038078 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:28.038028    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zxjft" podStartSLOduration=252.257278278 podStartE2EDuration="4m13.038011808s" totalImagesPullingTime="780.73353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:28.037033927 +0000 UTC m=+253.564815750" watchObservedRunningTime="2026-09-17 17:39:28.038011808 +0000 UTC m=+253.565793677"
Sep 17 17:39:28.144187 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:28.144150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:28.216168 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:28.216138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:31.045326 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:31.045267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:31.145869 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:31.145830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:32.193014 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:32.192982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:32.483986 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:32.483902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:33.177000 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.176966    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:39:33.177263 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.177242    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="alertmanager" containerID="cri-o://d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c" gracePeriod=120
Sep 17 17:39:33.177324 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.177303    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-metric" containerID="cri-o://13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff" gracePeriod=120
Sep 17 17:39:33.177391 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.177315    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-web" containerID="cri-o://0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b" gracePeriod=120
Sep 17 17:39:33.177391 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.177336    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy" containerID="cri-o://6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae" gracePeriod=120
Sep 17 17:39:33.177537 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.177387    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="config-reloader" containerID="cri-o://bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5" gracePeriod=120
Sep 17 17:39:33.177537 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:33.177421    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="prom-label-proxy" containerID="cri-o://2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff" gracePeriod=120
Sep 17 17:39:34.048316 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048284    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff" exitCode=0
Sep 17 17:39:34.048316 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048309    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff" exitCode=0
Sep 17 17:39:34.048316 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048316    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae" exitCode=0
Sep 17 17:39:34.048316 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048322    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5" exitCode=0
Sep 17 17:39:34.048812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048330    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c" exitCode=0
Sep 17 17:39:34.048812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff"}
Sep 17 17:39:34.048812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff"}
Sep 17 17:39:34.048812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae"}
Sep 17 17:39:34.048812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5"}
Sep 17 17:39:34.048812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.048493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c"}
Sep 17 17:39:34.419464 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.419438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:34.504465 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-web\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504530    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-main-db\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-cluster-tls-config\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-volume\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.504857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-web-config\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.505061 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504781    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-out\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.505061 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504913    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-metrics-client-ca\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.505061 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tlvk\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-kube-api-access-2tlvk\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.505061 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.504963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-tls-assets\") pod \"0e94cce6-f40c-47df-9f51-2c5013e8b641\" (UID: \"0e94cce6-f40c-47df-9f51-2c5013e8b641\") "
Sep 17 17:39:34.505061 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.505023    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:34.505319 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.505251    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.506029 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.505844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:34.506029 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.505987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:34.507611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.507552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.507981 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.507952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-volume" (OuterVolumeSpecName: "config-volume") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.508397 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.508367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.509203 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.509167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:34.509565 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.509512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-kube-api-access-2tlvk" (OuterVolumeSpecName: "kube-api-access-2tlvk") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "kube-api-access-2tlvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:34.509656 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.509612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-out" (OuterVolumeSpecName: "config-out") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:39:34.509786 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.509764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.509857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.509807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.513539 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.513413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.521030 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.520988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-web-config" (OuterVolumeSpecName: "web-config") pod "0e94cce6-f40c-47df-9f51-2c5013e8b641" (UID: "0e94cce6-f40c-47df-9f51-2c5013e8b641"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606382    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-main-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606428    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-alertmanager-main-db\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606438    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-cluster-tls-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606450    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-volume\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606459    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606468    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-web-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606477    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e94cce6-f40c-47df-9f51-2c5013e8b641-config-out\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606484    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e94cce6-f40c-47df-9f51-2c5013e8b641-metrics-client-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606491    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tlvk\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-kube-api-access-2tlvk\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606499    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e94cce6-f40c-47df-9f51-2c5013e8b641-tls-assets\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606916 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606509    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:34.606916 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:34.606519    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e94cce6-f40c-47df-9f51-2c5013e8b641-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:39:35.055586 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.055548    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerID="0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b" exitCode=0
Sep 17 17:39:35.055994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.055626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b"}
Sep 17 17:39:35.055994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.055653    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.055994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.055688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e94cce6-f40c-47df-9f51-2c5013e8b641","Type":"ContainerDied","Data":"f80158b580cde4a09d0c9c995c483a01c6a83a8f8193eeb510fdaff2c7a91e52"}
Sep 17 17:39:35.055994 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.055722    2563 scope.go:117] "RemoveContainer" containerID="2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff"
Sep 17 17:39:35.063186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.063164    2563 scope.go:117] "RemoveContainer" containerID="13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff"
Sep 17 17:39:35.070295 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.070277    2563 scope.go:117] "RemoveContainer" containerID="6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae"
Sep 17 17:39:35.077037 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.077010    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:39:35.078236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.078211    2563 scope.go:117] "RemoveContainer" containerID="0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b"
Sep 17 17:39:35.080890 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.080863    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:39:35.085629 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.085607    2563 scope.go:117] "RemoveContainer" containerID="bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5"
Sep 17 17:39:35.092924 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.092906    2563 scope.go:117] "RemoveContainer" containerID="d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c"
Sep 17 17:39:35.100151 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.100129    2563 scope.go:117] "RemoveContainer" containerID="94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29"
Sep 17 17:39:35.107500 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.107481    2563 scope.go:117] "RemoveContainer" containerID="2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff"
Sep 17 17:39:35.107904 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.107872    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff\": container with ID starting with 2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff not found: ID does not exist" containerID="2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff"
Sep 17 17:39:35.107966 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.107913    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff"} err="failed to get container status \"2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff\": rpc error: code = NotFound desc = could not find container \"2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff\": container with ID starting with 2c5a6d8aec4b65e8d8dd08e0c9452fff0b5a663001055ada9459875c572cf9ff not found: ID does not exist"
Sep 17 17:39:35.107966 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.107934    2563 scope.go:117] "RemoveContainer" containerID="13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff"
Sep 17 17:39:35.108335 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.108312    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff\": container with ID starting with 13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff not found: ID does not exist" containerID="13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff"
Sep 17 17:39:35.108407 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.108341    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff"} err="failed to get container status \"13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff\": rpc error: code = NotFound desc = could not find container \"13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff\": container with ID starting with 13234a800710e66444f92de34834dfa343ba881572c1294c629ad2077a4569ff not found: ID does not exist"
Sep 17 17:39:35.108407 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.108359    2563 scope.go:117] "RemoveContainer" containerID="6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae"
Sep 17 17:39:35.108622 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.108605    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae\": container with ID starting with 6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae not found: ID does not exist" containerID="6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae"
Sep 17 17:39:35.108661 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.108626    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae"} err="failed to get container status \"6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae\": rpc error: code = NotFound desc = could not find container \"6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae\": container with ID starting with 6364ec5f8e9779c36db6e969ab3f69f9916c0bfe2a651320e5ea17eb723234ae not found: ID does not exist"
Sep 17 17:39:35.108661 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.108639    2563 scope.go:117] "RemoveContainer" containerID="0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b"
Sep 17 17:39:35.108904 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.108883    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b\": container with ID starting with 0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b not found: ID does not exist" containerID="0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b"
Sep 17 17:39:35.108992 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.108922    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b"} err="failed to get container status \"0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b\": rpc error: code = NotFound desc = could not find container \"0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b\": container with ID starting with 0375b50a3fe7be2213f70ece9890c452f3122b13a138372361406b464b36d76b not found: ID does not exist"
Sep 17 17:39:35.108992 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.108938    2563 scope.go:117] "RemoveContainer" containerID="bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5"
Sep 17 17:39:35.109192 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.109174    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5\": container with ID starting with bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5 not found: ID does not exist" containerID="bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5"
Sep 17 17:39:35.109236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.109198    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5"} err="failed to get container status \"bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5\": rpc error: code = NotFound desc = could not find container \"bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5\": container with ID starting with bac4fee31409d34c4c3bbb4e8137a41ae113bd875798e94f43c1ca2426a75ad5 not found: ID does not exist"
Sep 17 17:39:35.109236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.109217    2563 scope.go:117] "RemoveContainer" containerID="d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c"
Sep 17 17:39:35.109446 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.109426    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c\": container with ID starting with d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c not found: ID does not exist" containerID="d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c"
Sep 17 17:39:35.109511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.109451    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c"} err="failed to get container status \"d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c\": rpc error: code = NotFound desc = could not find container \"d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c\": container with ID starting with d81dc3c24b0e6986f011ed26e9867c7e5942330e82a435c27c28a34b94879c2c not found: ID does not exist"
Sep 17 17:39:35.109511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.109465    2563 scope.go:117] "RemoveContainer" containerID="94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29"
Sep 17 17:39:35.109685 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:39:35.109666    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29\": container with ID starting with 94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29 not found: ID does not exist" containerID="94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29"
Sep 17 17:39:35.109740 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.109693    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29"} err="failed to get container status \"94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29\": rpc error: code = NotFound desc = could not find container \"94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29\": container with ID starting with 94f5405cf4cc8f61a12cf4d40801812cd7a318642383649f28a6cf422ad43a29 not found: ID does not exist"
Sep 17 17:39:35.117571 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117548    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:39:35.117832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117820    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-metric"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117834    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-metric"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c5cf63-5bbf-41f8-96d6-7860724fe3e3" containerName="console"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117849    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c5cf63-5bbf-41f8-96d6-7860724fe3e3" containerName="console"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117859    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117875    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="config-reloader"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="config-reloader"
Sep 17 17:39:35.117885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117886    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adcc76e6-46f8-4af7-8058-e20b872b35f7" containerName="console"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="adcc76e6-46f8-4af7-8058-e20b872b35f7" containerName="console"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="init-config-reloader"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117901    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="init-config-reloader"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117910    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="alertmanager"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117915    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="alertmanager"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117920    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-web"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117926    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-web"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117941    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="prom-label-proxy"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="prom-label-proxy"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117961    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686" containerName="registry"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.117969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686" containerName="registry"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118027    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="alertmanager"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118039    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c5cf63-5bbf-41f8-96d6-7860724fe3e3" containerName="console"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118046    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118053    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="config-reloader"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="prom-label-proxy"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118064    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb44e32-45d0-4fde-b7cf-0384ab5f1686" containerName="registry"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118069    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-metric"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118076    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="adcc76e6-46f8-4af7-8058-e20b872b35f7" containerName="console"
Sep 17 17:39:35.118158 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.118085    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" containerName="kube-rbac-proxy-web"
Sep 17 17:39:35.121698 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.121681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.124251 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124231    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:39:35.124342 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:39:35.124342 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124231    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:39:35.124481 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qr2b\""
Sep 17 17:39:35.124599 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:39:35.124677 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:39:35.124759 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124712    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:39:35.124759 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.124735    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:39:35.125060 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.125043    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:39:35.129692 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.129671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:39:35.150050 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.150020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:39:35.210012 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.209973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210012 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-web-config\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-config-out\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210196 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210153    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2np7\" (UniqueName: \"kubernetes.io/projected/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-kube-api-access-c2np7\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.210382 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.210328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311606 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-web-config\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-config-out\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2np7\" (UniqueName: \"kubernetes.io/projected/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-kube-api-access-c2np7\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.311832 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.312274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.312274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.312274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.312274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.311928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.312274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.312124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.313081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.312706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.313081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.313037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.314952 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.314931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-web-config\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315411 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315493 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315544 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315589 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315654 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315654 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.315988 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.315972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-config-out\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.316469 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.316451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.327173 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.327149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2np7\" (UniqueName: \"kubernetes.io/projected/bd8e401e-106b-4959-a7d9-76ae8b9c4b4a-kube-api-access-c2np7\") pod \"alertmanager-main-0\" (UID: \"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.432075 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.432029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:39:35.712624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:35.712453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:39:35.714739 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:39:35.714709    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8e401e_106b_4959_a7d9_76ae8b9c4b4a.slice/crio-683d93cb5dd8d730b7a8847ef835f08a1b3ed1572b796cd66475f53699f3fcc6 WatchSource:0}: Error finding container 683d93cb5dd8d730b7a8847ef835f08a1b3ed1572b796cd66475f53699f3fcc6: Status 404 returned error can't find the container with id 683d93cb5dd8d730b7a8847ef835f08a1b3ed1572b796cd66475f53699f3fcc6
Sep 17 17:39:36.061593 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:36.061509    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd8e401e-106b-4959-a7d9-76ae8b9c4b4a" containerID="73a6c6a161d1057e32bc50b4aee70d694a89c0fdcb7aa3d083059aed32f0f5de" exitCode=0
Sep 17 17:39:36.061593 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:36.061567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerDied","Data":"73a6c6a161d1057e32bc50b4aee70d694a89c0fdcb7aa3d083059aed32f0f5de"}
Sep 17 17:39:36.062007 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:36.061616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"683d93cb5dd8d730b7a8847ef835f08a1b3ed1572b796cd66475f53699f3fcc6"}
Sep 17 17:39:36.957431 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:36.957394    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e94cce6-f40c-47df-9f51-2c5013e8b641" path="/var/lib/kubelet/pods/0e94cce6-f40c-47df-9f51-2c5013e8b641/volumes"
Sep 17 17:39:37.072849 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.072816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"d02a2e11067957203a0a7d8faefe853ff7bc1a51ec24ee7b0b23b7ccc8aaa39e"}
Sep 17 17:39:37.073228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.072878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"1175b786d0be4d697de348de456c986f11b6802fe37ba7398a928452b5676d00"}
Sep 17 17:39:37.073228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.072906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"43aa8c3d8ae0ae8e91683b4becab3b8ca48774ffef89fbbbb68dbda4beeffec1"}
Sep 17 17:39:37.073228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.072930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"e2d08c43785dd6da5f9870b3c231a28220c27129082ecec1239c6564ed39bed8"}
Sep 17 17:39:37.073228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.072953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"9596d928022893f7eb1b59cc6fde716fa4a026147d934ef958e61a9a8139c66a"}
Sep 17 17:39:37.073228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.072975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd8e401e-106b-4959-a7d9-76ae8b9c4b4a","Type":"ContainerStarted","Data":"dcb091981d9bdff9dfbd78b668b8589f827e414d3406e0e417ae43f9cdfa7771"}
Sep 17 17:39:37.102924 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.102870    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.085911606 podStartE2EDuration="2.102849112s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.937506ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:37.101663874 +0000 UTC m=+262.629445699" watchObservedRunningTime="2026-09-17 17:39:37.102849112 +0000 UTC m=+262.630630937"
Sep 17 17:39:37.346229 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.346139    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-556cd5b59-pm54b"]
Sep 17 17:39:37.348496 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.348472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.351549 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.351525    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:39:37.351549 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.351544    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:39:37.351720 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.351593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-22qk2\""
Sep 17 17:39:37.351999 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.351982    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:39:37.352114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.352029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:39:37.352114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.352063    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:39:37.361948 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.361920    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:39:37.372250 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.372217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-556cd5b59-pm54b"]
Sep 17 17:39:37.430547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-serving-certs-ca-bundle\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-metrics-client-ca\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430782 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-secret-telemeter-client\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430782 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-telemeter-client-tls\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430782 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430782 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430782 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstqs\" (UniqueName: \"kubernetes.io/projected/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-kube-api-access-xstqs\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.430782 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.430815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-federate-client-tls\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531722 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-secret-telemeter-client\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531722 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-telemeter-client-tls\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531974 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531974 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531974 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstqs\" (UniqueName: \"kubernetes.io/projected/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-kube-api-access-xstqs\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531974 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-federate-client-tls\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531974 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-serving-certs-ca-bundle\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.531974 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.531972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-metrics-client-ca\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.532734 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.532704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-metrics-client-ca\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.532880 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.532758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.533083 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.533064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-serving-certs-ca-bundle\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.534274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.534252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-telemeter-client-tls\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.534575 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.534548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-federate-client-tls\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.534719 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.534700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.534774 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.534749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-secret-telemeter-client\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.540572 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.540549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstqs\" (UniqueName: \"kubernetes.io/projected/f68a0ad2-254b-43b9-809b-da9cfb8ee1e0-kube-api-access-xstqs\") pod \"telemeter-client-556cd5b59-pm54b\" (UID: \"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0\") " pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.658784 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.658703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b"
Sep 17 17:39:37.934532 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:37.934500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-556cd5b59-pm54b"]
Sep 17 17:39:37.937575 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:39:37.937543    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68a0ad2_254b_43b9_809b_da9cfb8ee1e0.slice/crio-22cce0468e5574184e6f66993a88b4a12b5d7272daf4bdcff910275752a67cf8 WatchSource:0}: Error finding container 22cce0468e5574184e6f66993a88b4a12b5d7272daf4bdcff910275752a67cf8: Status 404 returned error can't find the container with id 22cce0468e5574184e6f66993a88b4a12b5d7272daf4bdcff910275752a67cf8
Sep 17 17:39:38.078069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:38.078037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b" event={"ID":"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0","Type":"ContainerStarted","Data":"22cce0468e5574184e6f66993a88b4a12b5d7272daf4bdcff910275752a67cf8"}
Sep 17 17:39:40.089411 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.089368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b" event={"ID":"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0","Type":"ContainerStarted","Data":"7a5ea842fad9bd0ac078e4d9734cc57219bbc5f359beef133fb12781e055ffe3"}
Sep 17 17:39:40.089902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.089439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b" event={"ID":"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0","Type":"ContainerStarted","Data":"978f7e8884f2023147e0c1e9bdac2a8c149f3b29ebcb1b20d1b002a7a7edc72b"}
Sep 17 17:39:40.089902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.089467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b" event={"ID":"f68a0ad2-254b-43b9-809b-da9cfb8ee1e0","Type":"ContainerStarted","Data":"680b126df6f300e497cd16cc220e5995ca307d7c68e53f43d772c021e4fe6baa"}
Sep 17 17:39:40.120421 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.120365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-556cd5b59-pm54b" podStartSLOduration=1.5078780090000001 podStartE2EDuration="3.120348296s" totalImagesPullingTime="1.612470287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:40.117971265 +0000 UTC m=+265.645753085" watchObservedRunningTime="2026-09-17 17:39:40.120348296 +0000 UTC m=+265.648130121"
Sep 17 17:39:40.283902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.283871    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59656f8c7f-9sbsr"]
Sep 17 17:39:40.286576 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.286167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.353518 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.353432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59656f8c7f-9sbsr"]
Sep 17 17:39:40.357881 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.357847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-config\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.358048 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.357892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-trusted-ca-bundle\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.358048 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.357913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-service-ca\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.358048 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.358032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-serving-cert\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.358206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.358109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbd9\" (UniqueName: \"kubernetes.io/projected/0ef23821-1935-4e7d-8457-ea5a12f689f2-kube-api-access-gpbd9\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.358206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.358143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-oauth-config\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.358206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.358177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-oauth-serving-cert\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.397422 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.397390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:40.459096 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbd9\" (UniqueName: \"kubernetes.io/projected/0ef23821-1935-4e7d-8457-ea5a12f689f2-kube-api-access-gpbd9\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.459096 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-oauth-config\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.459351 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-oauth-serving-cert\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.459351 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-config\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.459351 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-trusted-ca-bundle\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.459351 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-service-ca\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.459520 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.459357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-serving-cert\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.460040 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.460011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-service-ca\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.460040 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.460034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-oauth-serving-cert\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.460201 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.460170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-trusted-ca-bundle\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.460426 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.460397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-config\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.461821 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.461781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-oauth-config\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.461897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.461843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-serving-cert\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.470091 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.470068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbd9\" (UniqueName: \"kubernetes.io/projected/0ef23821-1935-4e7d-8457-ea5a12f689f2-kube-api-access-gpbd9\") pod \"console-59656f8c7f-9sbsr\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") " pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.484132 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.484103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:40.596460 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.596416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:40.877399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:40.877324    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59656f8c7f-9sbsr"]
Sep 17 17:39:40.880536 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:39:40.880502    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef23821_1935_4e7d_8457_ea5a12f689f2.slice/crio-7b7e27df9cfef5411bf56691e8e497465db44118c74e02cf190ec6eca63298ba WatchSource:0}: Error finding container 7b7e27df9cfef5411bf56691e8e497465db44118c74e02cf190ec6eca63298ba: Status 404 returned error can't find the container with id 7b7e27df9cfef5411bf56691e8e497465db44118c74e02cf190ec6eca63298ba
Sep 17 17:39:41.094623 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:41.094578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59656f8c7f-9sbsr" event={"ID":"0ef23821-1935-4e7d-8457-ea5a12f689f2","Type":"ContainerStarted","Data":"e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e"}
Sep 17 17:39:41.095011 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:41.094643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59656f8c7f-9sbsr" event={"ID":"0ef23821-1935-4e7d-8457-ea5a12f689f2","Type":"ContainerStarted","Data":"7b7e27df9cfef5411bf56691e8e497465db44118c74e02cf190ec6eca63298ba"}
Sep 17 17:39:41.115166 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:41.115104    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59656f8c7f-9sbsr" podStartSLOduration=1.115088689 podStartE2EDuration="1.115088689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:41.113096138 +0000 UTC m=+266.640877963" watchObservedRunningTime="2026-09-17 17:39:41.115088689 +0000 UTC m=+266.642870514"
Sep 17 17:39:50.597539 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:50.597499    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:50.597936 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:50.597587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:50.602335 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:50.602309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:51.139601 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:51.139573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:39:51.200051 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:51.200015    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b8dbf9ff-zqcfp"]
Sep 17 17:39:51.317924 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:51.317892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:51.419531 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:51.419445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:51.519940 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:51.519903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:55.527645 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:55.527607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:55.614997 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:55.614966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:56.929378 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:39:56.929338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:07.755826 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:07.755775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:07.821374 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:07.821338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:14.806914 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:14.806887    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:40:14.811543 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:14.811512    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:40:14.817676 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:14.817651    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:40:16.224429 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.224366    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58b8dbf9ff-zqcfp" podUID="b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" containerName="console" containerID="cri-o://52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d" gracePeriod=15
Sep 17 17:40:16.474485 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.474421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:40:16.567534 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-service-ca\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.567712 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-trusted-ca-bundle\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.567712 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-oauth-serving-cert\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.567784 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-oauth-config\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.567784 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hmd9\" (UniqueName: \"kubernetes.io/projected/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-kube-api-access-2hmd9\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.567928 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-serving-cert\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.567928 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.567894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-config\") pod \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\" (UID: \"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd\") "
Sep 17 17:40:16.568132 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.568048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:16.568132 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.568072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:16.568396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.568171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:16.568396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.568186    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-trusted-ca-bundle\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:16.568396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.568207    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-service-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:16.568505 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.568410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-config" (OuterVolumeSpecName: "console-config") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:40:16.569945 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.569924    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-kube-api-access-2hmd9" (OuterVolumeSpecName: "kube-api-access-2hmd9") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "kube-api-access-2hmd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:40:16.570043 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.570023    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:40:16.570095 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.570042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" (UID: "b6b78e4f-fa8f-4acd-8451-aa659cf62dbd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:40:16.669240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.669201    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:16.669240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.669235    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:16.669240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.669249    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-oauth-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:16.669459 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.669261    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-console-oauth-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:16.669459 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:16.669273    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmd9\" (UniqueName: \"kubernetes.io/projected/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd-kube-api-access-2hmd9\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:40:17.238670 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.238575    2563 generic.go:356] "Generic (PLEG): container finished" podID="b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" containerID="52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d" exitCode=0
Sep 17 17:40:17.238670 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.238624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b8dbf9ff-zqcfp"
Sep 17 17:40:17.239195 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.238666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b8dbf9ff-zqcfp" event={"ID":"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd","Type":"ContainerDied","Data":"52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d"}
Sep 17 17:40:17.239195 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.238723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b8dbf9ff-zqcfp" event={"ID":"b6b78e4f-fa8f-4acd-8451-aa659cf62dbd","Type":"ContainerDied","Data":"d22a415b205caeab2e3328b08296e34665ef55b87936679a725390ba7d19f26a"}
Sep 17 17:40:17.239195 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.238749    2563 scope.go:117] "RemoveContainer" containerID="52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d"
Sep 17 17:40:17.247034 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.247015    2563 scope.go:117] "RemoveContainer" containerID="52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d"
Sep 17 17:40:17.247379 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:40:17.247358    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d\": container with ID starting with 52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d not found: ID does not exist" containerID="52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d"
Sep 17 17:40:17.247438 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.247389    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d"} err="failed to get container status \"52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d\": rpc error: code = NotFound desc = could not find container \"52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d\": container with ID starting with 52cfcd025f42f44798e788cc645a4462d29fd660be4461fb307dd4d9fef7a96d not found: ID does not exist"
Sep 17 17:40:17.258341 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.258305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b8dbf9ff-zqcfp"]
Sep 17 17:40:17.260981 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:17.260955    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58b8dbf9ff-zqcfp"]
Sep 17 17:40:18.958000 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:18.957969    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" path="/var/lib/kubelet/pods/b6b78e4f-fa8f-4acd-8451-aa659cf62dbd/volumes"
Sep 17 17:40:47.438777 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:47.438741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:47.523097 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:47.523060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:56.643786 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.643744    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-767bbb8964-tn2bn"]
Sep 17 17:40:56.644219 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.644198    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" containerName="console"
Sep 17 17:40:56.644281 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.644221    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" containerName="console"
Sep 17 17:40:56.644281 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.644277    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b78e4f-fa8f-4acd-8451-aa659cf62dbd" containerName="console"
Sep 17 17:40:56.647524 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.647499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.696899 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.696863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-trusted-ca-bundle\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.696899 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.696901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-config\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.697098 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.696929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-oauth-config\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.697098 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.696999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-oauth-serving-cert\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.697098 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.697047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfzt\" (UniqueName: \"kubernetes.io/projected/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-kube-api-access-vmfzt\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.697098 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.697090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-service-ca\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.697226 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.697117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-serving-cert\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.708022 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.707990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767bbb8964-tn2bn"]
Sep 17 17:40:56.732967 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.732927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:56.797977 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.797943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-oauth-serving-cert\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.797995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfzt\" (UniqueName: \"kubernetes.io/projected/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-kube-api-access-vmfzt\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-service-ca\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-serving-cert\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-trusted-ca-bundle\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798363 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-config\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798363 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-oauth-config\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-oauth-serving-cert\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.798922 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-service-ca\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.799024 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.798999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-config\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.799089 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.799007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-trusted-ca-bundle\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.800661 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.800639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-serving-cert\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.800741 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.800712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-oauth-config\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.806309 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.806287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfzt\" (UniqueName: \"kubernetes.io/projected/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-kube-api-access-vmfzt\") pod \"console-767bbb8964-tn2bn\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") " pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:56.832173 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.832141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:56.904057 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.903965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:56.957908 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:56.957877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:40:57.238775 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:57.238736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767bbb8964-tn2bn"]
Sep 17 17:40:57.241876 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:40:57.241850    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede8eb3b_75ec_4141_91b6_c3ee3318efa2.slice/crio-2680aa417f41d20672c2e29261532341c257b8d6b49e6520e2040d8a94f6ee19 WatchSource:0}: Error finding container 2680aa417f41d20672c2e29261532341c257b8d6b49e6520e2040d8a94f6ee19: Status 404 returned error can't find the container with id 2680aa417f41d20672c2e29261532341c257b8d6b49e6520e2040d8a94f6ee19
Sep 17 17:40:57.400913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:57.400831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bbb8964-tn2bn" event={"ID":"ede8eb3b-75ec-4141-91b6-c3ee3318efa2","Type":"ContainerStarted","Data":"06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210"}
Sep 17 17:40:57.400913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:57.400882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bbb8964-tn2bn" event={"ID":"ede8eb3b-75ec-4141-91b6-c3ee3318efa2","Type":"ContainerStarted","Data":"2680aa417f41d20672c2e29261532341c257b8d6b49e6520e2040d8a94f6ee19"}
Sep 17 17:40:57.423695 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:57.423640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-767bbb8964-tn2bn" podStartSLOduration=1.423620955 podStartE2EDuration="1.423620955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:57.421619268 +0000 UTC m=+342.949401109" watchObservedRunningTime="2026-09-17 17:40:57.423620955 +0000 UTC m=+342.951402780"
Sep 17 17:40:57.440730 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:57.440677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:57.542127 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:57.542044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:59.545625 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:59.545592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:59.660540 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:40:59.660504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.346076 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:02.346042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.500262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:02.500229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:02.604757 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:02.604674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:06.958490 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:06.958457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:41:06.958913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:06.958511    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:41:06.963367 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:06.963339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:41:07.446483 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:07.446453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:41:07.515555 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:07.515518    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59656f8c7f-9sbsr"]
Sep 17 17:41:07.627988 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:07.627954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:07.729604 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:07.729515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:07.796611 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:07.796571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:32.538130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.538063    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59656f8c7f-9sbsr" podUID="0ef23821-1935-4e7d-8457-ea5a12f689f2" containerName="console" containerID="cri-o://e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e" gracePeriod=15
Sep 17 17:41:32.781256 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.781227    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:41:32.902514 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-config\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.902514 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-service-ca\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.902514 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpbd9\" (UniqueName: \"kubernetes.io/projected/0ef23821-1935-4e7d-8457-ea5a12f689f2-kube-api-access-gpbd9\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.902767 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-trusted-ca-bundle\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.902767 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-oauth-config\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.902767 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-oauth-serving-cert\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.902767 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-serving-cert\") pod \"0ef23821-1935-4e7d-8457-ea5a12f689f2\" (UID: \"0ef23821-1935-4e7d-8457-ea5a12f689f2\") "
Sep 17 17:41:32.903015 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-service-ca" (OuterVolumeSpecName: "service-ca") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:32.903015 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.902901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-config" (OuterVolumeSpecName: "console-config") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:32.903124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.903090    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:32.903124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.903108    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-service-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:32.903206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.903131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:32.903206 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.903147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:41:32.904850 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.904818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:41:32.904850 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.904839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:41:32.905016 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:32.904931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef23821-1935-4e7d-8457-ea5a12f689f2-kube-api-access-gpbd9" (OuterVolumeSpecName: "kube-api-access-gpbd9") pod "0ef23821-1935-4e7d-8457-ea5a12f689f2" (UID: "0ef23821-1935-4e7d-8457-ea5a12f689f2"). InnerVolumeSpecName "kube-api-access-gpbd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:33.003612 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.003582    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbd9\" (UniqueName: \"kubernetes.io/projected/0ef23821-1935-4e7d-8457-ea5a12f689f2-kube-api-access-gpbd9\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:33.003612 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.003609    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-trusted-ca-bundle\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:33.003612 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.003619    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-oauth-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:33.003868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.003629    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ef23821-1935-4e7d-8457-ea5a12f689f2-oauth-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:33.003868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.003638    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ef23821-1935-4e7d-8457-ea5a12f689f2-console-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:41:33.545420 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.545380    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ef23821-1935-4e7d-8457-ea5a12f689f2" containerID="e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e" exitCode=0
Sep 17 17:41:33.545822 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.545431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59656f8c7f-9sbsr"
Sep 17 17:41:33.545822 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.545471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59656f8c7f-9sbsr" event={"ID":"0ef23821-1935-4e7d-8457-ea5a12f689f2","Type":"ContainerDied","Data":"e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e"}
Sep 17 17:41:33.545822 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.545526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59656f8c7f-9sbsr" event={"ID":"0ef23821-1935-4e7d-8457-ea5a12f689f2","Type":"ContainerDied","Data":"7b7e27df9cfef5411bf56691e8e497465db44118c74e02cf190ec6eca63298ba"}
Sep 17 17:41:33.545822 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.545552    2563 scope.go:117] "RemoveContainer" containerID="e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e"
Sep 17 17:41:33.553837 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.553821    2563 scope.go:117] "RemoveContainer" containerID="e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e"
Sep 17 17:41:33.554102 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:41:33.554080    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e\": container with ID starting with e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e not found: ID does not exist" containerID="e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e"
Sep 17 17:41:33.554156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.554110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e"} err="failed to get container status \"e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e\": rpc error: code = NotFound desc = could not find container \"e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e\": container with ID starting with e7a53679c631ad2585d899c67035da8e81e593316ccaf592fad163cbd38e4c5e not found: ID does not exist"
Sep 17 17:41:33.563134 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.563110    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59656f8c7f-9sbsr"]
Sep 17 17:41:33.569103 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:33.569081    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59656f8c7f-9sbsr"]
Sep 17 17:41:34.958243 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:41:34.958208    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef23821-1935-4e7d-8457-ea5a12f689f2" path="/var/lib/kubelet/pods/0ef23821-1935-4e7d-8457-ea5a12f689f2/volumes"
Sep 17 17:42:05.010440 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.010402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.107157 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.107117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.162388 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.162354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:05.992075 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.992037    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"]
Sep 17 17:42:05.992371 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.992358    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef23821-1935-4e7d-8457-ea5a12f689f2" containerName="console"
Sep 17 17:42:05.992413 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.992374    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef23821-1935-4e7d-8457-ea5a12f689f2" containerName="console"
Sep 17 17:42:05.992446 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.992438    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef23821-1935-4e7d-8457-ea5a12f689f2" containerName="console"
Sep 17 17:42:05.995658 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.995638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:05.998052 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.998027    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:42:05.998142 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.998027    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:42:05.998737 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:05.998716    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8jhrx\""
Sep 17 17:42:06.003975 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.003951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"]
Sep 17 17:42:06.075764 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.075722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.076192 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.075783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.076192 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.075847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8t5\" (UniqueName: \"kubernetes.io/projected/8807ad9e-02cc-4937-9b6f-45efb89adccc-kube-api-access-hc8t5\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.176755 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.176714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8t5\" (UniqueName: \"kubernetes.io/projected/8807ad9e-02cc-4937-9b6f-45efb89adccc-kube-api-access-hc8t5\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.176947 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.176764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.176947 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.176853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.177118 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.177099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.177194 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.177178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.185469 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.185443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8t5\" (UniqueName: \"kubernetes.io/projected/8807ad9e-02cc-4937-9b6f-45efb89adccc-kube-api-access-hc8t5\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.304973 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.304891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:06.575258 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.575225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"]
Sep 17 17:42:06.577922 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:42:06.577897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8807ad9e_02cc_4937_9b6f_45efb89adccc.slice/crio-f10cb2167492bd8e3865097a4aa57b6ffca810bd333b4c2eeadbafc31ad2d9e5 WatchSource:0}: Error finding container f10cb2167492bd8e3865097a4aa57b6ffca810bd333b4c2eeadbafc31ad2d9e5: Status 404 returned error can't find the container with id f10cb2167492bd8e3865097a4aa57b6ffca810bd333b4c2eeadbafc31ad2d9e5
Sep 17 17:42:06.579973 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.579954    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:42:06.675609 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:06.675574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd" event={"ID":"8807ad9e-02cc-4937-9b6f-45efb89adccc","Type":"ContainerStarted","Data":"f10cb2167492bd8e3865097a4aa57b6ffca810bd333b4c2eeadbafc31ad2d9e5"}
Sep 17 17:42:12.704512 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:12.704478    2563 generic.go:356] "Generic (PLEG): container finished" podID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerID="d0ea2f4c0fb2cf303f13e9d5851aaaace1784359b71aca15b918750ddfafcfdd" exitCode=0
Sep 17 17:42:12.704923 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:12.704549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd" event={"ID":"8807ad9e-02cc-4937-9b6f-45efb89adccc","Type":"ContainerDied","Data":"d0ea2f4c0fb2cf303f13e9d5851aaaace1784359b71aca15b918750ddfafcfdd"}
Sep 17 17:42:20.736709 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:20.736626    2563 generic.go:356] "Generic (PLEG): container finished" podID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerID="136802925d71c33d3a6451c76b581289c60bc3ada548ccc7e65ad5b8b0572761" exitCode=0
Sep 17 17:42:20.736709 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:20.736687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd" event={"ID":"8807ad9e-02cc-4937-9b6f-45efb89adccc","Type":"ContainerDied","Data":"136802925d71c33d3a6451c76b581289c60bc3ada548ccc7e65ad5b8b0572761"}
Sep 17 17:42:27.769193 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:27.769144    2563 generic.go:356] "Generic (PLEG): container finished" podID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerID="6ee918ca2a02e058e91e69475dae5216c9e18914fac963b6b605d4c0a8c1b1cf" exitCode=0
Sep 17 17:42:27.769633 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:27.769223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd" event={"ID":"8807ad9e-02cc-4937-9b6f-45efb89adccc","Type":"ContainerDied","Data":"6ee918ca2a02e058e91e69475dae5216c9e18914fac963b6b605d4c0a8c1b1cf"}
Sep 17 17:42:28.951534 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:28.951497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:29.086136 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.086051    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-bundle\") pod \"8807ad9e-02cc-4937-9b6f-45efb89adccc\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") "
Sep 17 17:42:29.086136 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.086086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-util\") pod \"8807ad9e-02cc-4937-9b6f-45efb89adccc\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") "
Sep 17 17:42:29.086337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.086155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc8t5\" (UniqueName: \"kubernetes.io/projected/8807ad9e-02cc-4937-9b6f-45efb89adccc-kube-api-access-hc8t5\") pod \"8807ad9e-02cc-4937-9b6f-45efb89adccc\" (UID: \"8807ad9e-02cc-4937-9b6f-45efb89adccc\") "
Sep 17 17:42:29.086836 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.086783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-bundle" (OuterVolumeSpecName: "bundle") pod "8807ad9e-02cc-4937-9b6f-45efb89adccc" (UID: "8807ad9e-02cc-4937-9b6f-45efb89adccc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:29.088383 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.088363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8807ad9e-02cc-4937-9b6f-45efb89adccc-kube-api-access-hc8t5" (OuterVolumeSpecName: "kube-api-access-hc8t5") pod "8807ad9e-02cc-4937-9b6f-45efb89adccc" (UID: "8807ad9e-02cc-4937-9b6f-45efb89adccc"). InnerVolumeSpecName "kube-api-access-hc8t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:42:29.092447 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.092423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-util" (OuterVolumeSpecName: "util") pod "8807ad9e-02cc-4937-9b6f-45efb89adccc" (UID: "8807ad9e-02cc-4937-9b6f-45efb89adccc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:42:29.187752 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.187720    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-bundle\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:42:29.187752 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.187749    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8807ad9e-02cc-4937-9b6f-45efb89adccc-util\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:42:29.187752 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.187757    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8t5\" (UniqueName: \"kubernetes.io/projected/8807ad9e-02cc-4937-9b6f-45efb89adccc-kube-api-access-hc8t5\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:42:29.779081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.779051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd"
Sep 17 17:42:29.779081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.779072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ahsppd" event={"ID":"8807ad9e-02cc-4937-9b6f-45efb89adccc","Type":"ContainerDied","Data":"f10cb2167492bd8e3865097a4aa57b6ffca810bd333b4c2eeadbafc31ad2d9e5"}
Sep 17 17:42:29.779305 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:29.779114    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f10cb2167492bd8e3865097a4aa57b6ffca810bd333b4c2eeadbafc31ad2d9e5"
Sep 17 17:42:32.905598 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:32.905563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:34.298237 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.298204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:34.924911 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.924870    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"]
Sep 17 17:42:34.925181 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925169    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="extract"
Sep 17 17:42:34.925227 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925182    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="extract"
Sep 17 17:42:34.925227 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925213    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="pull"
Sep 17 17:42:34.925227 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925218    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="pull"
Sep 17 17:42:34.925360 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="util"
Sep 17 17:42:34.925360 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925242    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="util"
Sep 17 17:42:34.925360 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.925307    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8807ad9e-02cc-4937-9b6f-45efb89adccc" containerName="extract"
Sep 17 17:42:34.927544 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.927523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:34.930148 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.930127    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-lnj2n\""
Sep 17 17:42:34.930533 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.930518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:42:34.930633 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.930614    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:42:34.930696 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.930653    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:42:34.936484 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.936457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1b75f659-c322-4399-a83e-84f48f921f1e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpgpp\" (UID: \"1b75f659-c322-4399-a83e-84f48f921f1e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:34.936588 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.936559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf7cq\" (UniqueName: \"kubernetes.io/projected/1b75f659-c322-4399-a83e-84f48f921f1e-kube-api-access-nf7cq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpgpp\" (UID: \"1b75f659-c322-4399-a83e-84f48f921f1e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:34.940700 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:34.940678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"]
Sep 17 17:42:35.037886 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.037840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7cq\" (UniqueName: \"kubernetes.io/projected/1b75f659-c322-4399-a83e-84f48f921f1e-kube-api-access-nf7cq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpgpp\" (UID: \"1b75f659-c322-4399-a83e-84f48f921f1e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:35.038123 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.037945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1b75f659-c322-4399-a83e-84f48f921f1e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpgpp\" (UID: \"1b75f659-c322-4399-a83e-84f48f921f1e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:35.040368 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.040346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/1b75f659-c322-4399-a83e-84f48f921f1e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpgpp\" (UID: \"1b75f659-c322-4399-a83e-84f48f921f1e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:35.046704 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.046681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7cq\" (UniqueName: \"kubernetes.io/projected/1b75f659-c322-4399-a83e-84f48f921f1e-kube-api-access-nf7cq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-kpgpp\" (UID: \"1b75f659-c322-4399-a83e-84f48f921f1e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:35.240085 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.239972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:35.526333 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.526306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"]
Sep 17 17:42:35.528537 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:42:35.528514    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b75f659_c322_4399_a83e_84f48f921f1e.slice/crio-514dac9ce10ccb7c9fb36bd74d8662bef69638c71230f5ef16ab68fc93aee785 WatchSource:0}: Error finding container 514dac9ce10ccb7c9fb36bd74d8662bef69638c71230f5ef16ab68fc93aee785: Status 404 returned error can't find the container with id 514dac9ce10ccb7c9fb36bd74d8662bef69638c71230f5ef16ab68fc93aee785
Sep 17 17:42:35.804721 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:35.804634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp" event={"ID":"1b75f659-c322-4399-a83e-84f48f921f1e","Type":"ContainerStarted","Data":"514dac9ce10ccb7c9fb36bd74d8662bef69638c71230f5ef16ab68fc93aee785"}
Sep 17 17:42:38.778500 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:38.778465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:38.820477 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:38.820430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp" event={"ID":"1b75f659-c322-4399-a83e-84f48f921f1e","Type":"ContainerStarted","Data":"995bd25fe532e1f4a8395d7ef99a33756cbb1f60d522d8426c6912c3a3aa3925"}
Sep 17 17:42:38.820638 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:38.820535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:42:38.843583 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:38.843525    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp" podStartSLOduration=2.106140684 podStartE2EDuration="4.843508955s" totalImagesPullingTime="2.737368271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:38.842351167 +0000 UTC m=+444.370132990" watchObservedRunningTime="2026-09-17 17:42:38.843508955 +0000 UTC m=+444.371290780"
Sep 17 17:42:39.045416 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.045333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.057853 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.057820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-mrlzq"]
Sep 17 17:42:39.060112 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.060074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.062584 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.062564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 17:42:39.062718 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.062598    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:42:39.062940 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.062925    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-kqm7q\""
Sep 17 17:42:39.073334 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.073303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-certificates\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.073478 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.073349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/993043f5-0621-4a3c-b0cb-3cec2551baaf-cabundle0\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.073478 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.073435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcmm\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-kube-api-access-xpcmm\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.073640 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.073622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-mrlzq"]
Sep 17 17:42:39.174099 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.174051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcmm\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-kube-api-access-xpcmm\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.174291 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.174124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-certificates\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.174291 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.174173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/993043f5-0621-4a3c-b0cb-3cec2551baaf-cabundle0\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.174537 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.174505    2563 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 17:42:39.174537 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.174533    2563 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:42:39.174712 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.174544    2563 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:42:39.174712 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.174559    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-mrlzq: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 17:42:39.174712 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.174630    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-certificates podName:993043f5-0621-4a3c-b0cb-3cec2551baaf nodeName:}" failed. No retries permitted until 2026-09-17 17:42:39.674612805 +0000 UTC m=+445.202394618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-certificates") pod "keda-operator-579c8958f7-mrlzq" (UID: "993043f5-0621-4a3c-b0cb-3cec2551baaf") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 17:42:39.174990 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.174969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/993043f5-0621-4a3c-b0cb-3cec2551baaf-cabundle0\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.187486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.187459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcmm\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-kube-api-access-xpcmm\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.329108 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.329012    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.527153 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.527116    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"]
Sep 17 17:42:39.529700 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.529681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.532407 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.532383    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 17:42:39.542095 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.542062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.542276 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.542236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"]
Sep 17 17:42:39.577230 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.577175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c2g\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-kube-api-access-48c2g\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.577406 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.577308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.577406 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.577338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.678754 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.678720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-certificates\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.678980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.678786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c2g\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-kube-api-access-48c2g\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.678980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.678892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.678980 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.678923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.679187 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.679040    2563 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 17:42:39.679187 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.679061    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn: secret "keda-metrics-apiserver-certs" not found
Sep 17 17:42:39.679187 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:39.679119    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates podName:4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d nodeName:}" failed. No retries permitted until 2026-09-17 17:42:40.179099684 +0000 UTC m=+445.706881494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates") pod "keda-metrics-apiserver-77857b7b98-g5gjn" (UID: "4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d") : secret "keda-metrics-apiserver-certs" not found
Sep 17 17:42:39.679577 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.679552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.682027 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.682001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/993043f5-0621-4a3c-b0cb-3cec2551baaf-certificates\") pod \"keda-operator-579c8958f7-mrlzq\" (UID: \"993043f5-0621-4a3c-b0cb-3cec2551baaf\") " pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.687584 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.687558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c2g\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-kube-api-access-48c2g\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:39.723719 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.723683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.793038 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.792998    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-87xrf"]
Sep 17 17:42:39.795348 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.795325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.797997 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.797975    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:42:39.812915 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.812883    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-87xrf"]
Sep 17 17:42:39.822944 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.822907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.881325 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.881286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0b65c0bf-aa10-4d30-b242-5ecaac7473f7-certificates\") pod \"keda-admission-6c4d864b86-87xrf\" (UID: \"0b65c0bf-aa10-4d30-b242-5ecaac7473f7\") " pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.881533 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.881346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zst9p\" (UniqueName: \"kubernetes.io/projected/0b65c0bf-aa10-4d30-b242-5ecaac7473f7-kube-api-access-zst9p\") pod \"keda-admission-6c4d864b86-87xrf\" (UID: \"0b65c0bf-aa10-4d30-b242-5ecaac7473f7\") " pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.893637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.893600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.970895 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.970771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:39.982240 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.982211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0b65c0bf-aa10-4d30-b242-5ecaac7473f7-certificates\") pod \"keda-admission-6c4d864b86-87xrf\" (UID: \"0b65c0bf-aa10-4d30-b242-5ecaac7473f7\") " pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.982344 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.982261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zst9p\" (UniqueName: \"kubernetes.io/projected/0b65c0bf-aa10-4d30-b242-5ecaac7473f7-kube-api-access-zst9p\") pod \"keda-admission-6c4d864b86-87xrf\" (UID: \"0b65c0bf-aa10-4d30-b242-5ecaac7473f7\") " pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.984704 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.984680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0b65c0bf-aa10-4d30-b242-5ecaac7473f7-certificates\") pod \"keda-admission-6c4d864b86-87xrf\" (UID: \"0b65c0bf-aa10-4d30-b242-5ecaac7473f7\") " pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.991458 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.991432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zst9p\" (UniqueName: \"kubernetes.io/projected/0b65c0bf-aa10-4d30-b242-5ecaac7473f7-kube-api-access-zst9p\") pod \"keda-admission-6c4d864b86-87xrf\" (UID: \"0b65c0bf-aa10-4d30-b242-5ecaac7473f7\") " pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:39.994435 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:39.994404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:40.107955 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.107916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:40.184236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.184205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:40.184411 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:40.184383    2563 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 17:42:40.184411 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:40.184405    2563 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn: secret "keda-metrics-apiserver-certs" not found
Sep 17 17:42:40.184512 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:42:40.184460    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates podName:4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d nodeName:}" failed. No retries permitted until 2026-09-17 17:42:41.184442555 +0000 UTC m=+446.712224360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates") pod "keda-metrics-apiserver-77857b7b98-g5gjn" (UID: "4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d") : secret "keda-metrics-apiserver-certs" not found
Sep 17 17:42:40.256573 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.256544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-mrlzq"]
Sep 17 17:42:40.257983 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:42:40.257950    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993043f5_0621_4a3c_b0cb_3cec2551baaf.slice/crio-62d3f3c14743b0e8ea4da64cd69fa990bb147be8433f2974d9fbdf8e240611db WatchSource:0}: Error finding container 62d3f3c14743b0e8ea4da64cd69fa990bb147be8433f2974d9fbdf8e240611db: Status 404 returned error can't find the container with id 62d3f3c14743b0e8ea4da64cd69fa990bb147be8433f2974d9fbdf8e240611db
Sep 17 17:42:40.363021 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.362989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:40.394002 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.393969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-87xrf"]
Sep 17 17:42:40.394134 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:42:40.394113    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b65c0bf_aa10_4d30_b242_5ecaac7473f7.slice/crio-745b931d31f7d6f3e03d52bf4baa54a1bc2c9aaa7a202be2e5c631ed3a817686 WatchSource:0}: Error finding container 745b931d31f7d6f3e03d52bf4baa54a1bc2c9aaa7a202be2e5c631ed3a817686: Status 404 returned error can't find the container with id 745b931d31f7d6f3e03d52bf4baa54a1bc2c9aaa7a202be2e5c631ed3a817686
Sep 17 17:42:40.465239 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.465189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:40.636284 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.636192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:40.737922 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.737881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:40.832316 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.832244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-mrlzq" event={"ID":"993043f5-0621-4a3c-b0cb-3cec2551baaf","Type":"ContainerStarted","Data":"62d3f3c14743b0e8ea4da64cd69fa990bb147be8433f2974d9fbdf8e240611db"}
Sep 17 17:42:40.834383 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.834346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-87xrf" event={"ID":"0b65c0bf-aa10-4d30-b242-5ecaac7473f7","Type":"ContainerStarted","Data":"745b931d31f7d6f3e03d52bf4baa54a1bc2c9aaa7a202be2e5c631ed3a817686"}
Sep 17 17:42:40.838078 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:40.838044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:41.195061 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:41.195018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:41.198699 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:41.198300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-g5gjn\" (UID: \"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:41.343159 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:41.343117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:41.654952 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:41.654922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"]
Sep 17 17:42:41.829903 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:42:41.829861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce1296c_2faf_46c0_bbc4_8c3c5fbe1c9d.slice/crio-cbb394558072077c50914217201d924f6e4c0770ae0e8d50ad773d50e8e1da18 WatchSource:0}: Error finding container cbb394558072077c50914217201d924f6e4c0770ae0e8d50ad773d50e8e1da18: Status 404 returned error can't find the container with id cbb394558072077c50914217201d924f6e4c0770ae0e8d50ad773d50e8e1da18
Sep 17 17:42:41.840212 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:41.840179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn" event={"ID":"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d","Type":"ContainerStarted","Data":"cbb394558072077c50914217201d924f6e4c0770ae0e8d50ad773d50e8e1da18"}
Sep 17 17:42:42.848260 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:42.848217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-87xrf" event={"ID":"0b65c0bf-aa10-4d30-b242-5ecaac7473f7","Type":"ContainerStarted","Data":"92a58bdd01cb1df2aa28547e82646eb1b598ad91ffba597fafe3477885a5eb7c"}
Sep 17 17:42:42.848712 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:42.848337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:42:42.865360 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:42.865176    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-87xrf" podStartSLOduration=2.3960858099999998 podStartE2EDuration="3.865154491s" totalImagesPullingTime="1.469068681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:42.864477178 +0000 UTC m=+448.392259004" watchObservedRunningTime="2026-09-17 17:42:42.865154491 +0000 UTC m=+448.392936327"
Sep 17 17:42:45.865408 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:45.865350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn" event={"ID":"4ce1296c-2faf-46c0-bbc4-8c3c5fbe1c9d","Type":"ContainerStarted","Data":"85c89604801f7d854763938d3267c1236d40aa4031b235db9993427c5dea74b9"}
Sep 17 17:42:45.865982 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:45.865452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:45.886888 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:45.886828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn" podStartSLOduration=3.872926148 podStartE2EDuration="6.886807206s" totalImagesPullingTime="3.013881058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:45.884708163 +0000 UTC m=+451.412489991" watchObservedRunningTime="2026-09-17 17:42:45.886807206 +0000 UTC m=+451.414589027"
Sep 17 17:42:46.872179 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:46.872137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-mrlzq" event={"ID":"993043f5-0621-4a3c-b0cb-3cec2551baaf","Type":"ContainerStarted","Data":"22bad54428a60200e956a53a8dc84ed4a768367519d34641cb45fdecd0c3da6c"}
Sep 17 17:42:46.872641 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:46.872309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:42:46.897431 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:46.897365    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-mrlzq" podStartSLOduration=2.074366883 podStartE2EDuration="7.897343082s" totalImagesPullingTime="5.822976199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:46.894182113 +0000 UTC m=+452.421963940" watchObservedRunningTime="2026-09-17 17:42:46.897343082 +0000 UTC m=+452.425124908"
Sep 17 17:42:56.876732 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:56.876702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-g5gjn"
Sep 17 17:42:59.827709 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:42:59.827679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-kpgpp"
Sep 17 17:43:00.324234 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:00.324202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:03.855114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:03.855085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-87xrf"
Sep 17 17:43:07.878730 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:07.878702    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-mrlzq"
Sep 17 17:43:16.506648 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.506614    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868fdb6c4c-8p26w"]
Sep 17 17:43:16.509125 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.509106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.521315 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.521286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868fdb6c4c-8p26w"]
Sep 17 17:43:16.559207 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.559176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:16.612730 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-trusted-ca-bundle\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.612857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-service-ca\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.612857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-serving-cert\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.612857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld75v\" (UniqueName: \"kubernetes.io/projected/1f6b9919-8efd-4253-aab7-0cce002ad3c9-kube-api-access-ld75v\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.612857 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-config\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.613057 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-oauth-config\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.613057 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.612932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-oauth-serving-cert\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.622281 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.622262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:16.713300 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-trusted-ca-bundle\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.713300 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-service-ca\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.713449 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-serving-cert\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.713485 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld75v\" (UniqueName: \"kubernetes.io/projected/1f6b9919-8efd-4253-aab7-0cce002ad3c9-kube-api-access-ld75v\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.713525 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-config\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.713566 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-oauth-config\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.713606 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.713567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-oauth-serving-cert\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.714069 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.714046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-service-ca\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.714227 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.714200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-config\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.714227 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.714216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-trusted-ca-bundle\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.714331 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.714225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f6b9919-8efd-4253-aab7-0cce002ad3c9-oauth-serving-cert\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.715916 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.715901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-oauth-config\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.716079 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.716064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f6b9919-8efd-4253-aab7-0cce002ad3c9-console-serving-cert\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.722471 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.722446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld75v\" (UniqueName: \"kubernetes.io/projected/1f6b9919-8efd-4253-aab7-0cce002ad3c9-kube-api-access-ld75v\") pod \"console-868fdb6c4c-8p26w\" (UID: \"1f6b9919-8efd-4253-aab7-0cce002ad3c9\") " pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:16.818344 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:16.818282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:17.094540 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:17.094516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868fdb6c4c-8p26w"]
Sep 17 17:43:17.096597 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:43:17.096552    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6b9919_8efd_4253_aab7_0cce002ad3c9.slice/crio-79397f0b05cc4da115a565c9b9316a921cff607130dd0f4996c79ae793edd09b WatchSource:0}: Error finding container 79397f0b05cc4da115a565c9b9316a921cff607130dd0f4996c79ae793edd09b: Status 404 returned error can't find the container with id 79397f0b05cc4da115a565c9b9316a921cff607130dd0f4996c79ae793edd09b
Sep 17 17:43:18.018697 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:18.018661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868fdb6c4c-8p26w" event={"ID":"1f6b9919-8efd-4253-aab7-0cce002ad3c9","Type":"ContainerStarted","Data":"498f0cf44c5075ede38e87c76ac1333c30baecfa69cb4f4700e0bc8ef4d791c2"}
Sep 17 17:43:18.019207 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:18.018714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868fdb6c4c-8p26w" event={"ID":"1f6b9919-8efd-4253-aab7-0cce002ad3c9","Type":"ContainerStarted","Data":"79397f0b05cc4da115a565c9b9316a921cff607130dd0f4996c79ae793edd09b"}
Sep 17 17:43:18.037732 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:18.037676    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868fdb6c4c-8p26w" podStartSLOduration=2.03765679 podStartE2EDuration="2.03765679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:18.037044672 +0000 UTC m=+483.564826513" watchObservedRunningTime="2026-09-17 17:43:18.03765679 +0000 UTC m=+483.565438617"
Sep 17 17:43:26.819133 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:26.819096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:26.819568 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:26.819384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:26.824062 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:26.824039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:27.062601 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:27.062569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868fdb6c4c-8p26w"
Sep 17 17:43:27.132810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:27.132706    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767bbb8964-tn2bn"]
Sep 17 17:43:27.245840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:27.245751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:27.356851 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:27.356814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:27.436843 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:27.436774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:31.213511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:31.213480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:31.314408 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:31.314370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:52.153523 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.153460    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-767bbb8964-tn2bn" podUID="ede8eb3b-75ec-4141-91b6-c3ee3318efa2" containerName="console" containerID="cri-o://06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210" gracePeriod=15
Sep 17 17:43:52.390066 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.390037    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:43:52.514841 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.514780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-service-ca\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.514841 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.514846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-serving-cert\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.515058 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.514896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-config\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.515058 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.514948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-oauth-config\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.515058 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-oauth-serving-cert\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.515058 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmfzt\" (UniqueName: \"kubernetes.io/projected/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-kube-api-access-vmfzt\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.515267 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-trusted-ca-bundle\") pod \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\" (UID: \"ede8eb3b-75ec-4141-91b6-c3ee3318efa2\") "
Sep 17 17:43:52.515330 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-service-ca" (OuterVolumeSpecName: "service-ca") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:52.515446 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-config" (OuterVolumeSpecName: "console-config") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:52.515513 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515468    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-service-ca\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.515513 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:52.515594 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.515546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:43:52.517192 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.517155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-kube-api-access-vmfzt" (OuterVolumeSpecName: "kube-api-access-vmfzt") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "kube-api-access-vmfzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:52.517380 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.517359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:52.517488 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.517473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ede8eb3b-75ec-4141-91b6-c3ee3318efa2" (UID: "ede8eb3b-75ec-4141-91b6-c3ee3318efa2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:52.616995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.616890    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.616995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.616934    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-oauth-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.616995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.616946    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-oauth-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.616995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.616958    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmfzt\" (UniqueName: \"kubernetes.io/projected/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-kube-api-access-vmfzt\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.616995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.616966    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-trusted-ca-bundle\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.616995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:52.616974    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ede8eb3b-75ec-4141-91b6-c3ee3318efa2-console-serving-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:43:53.187417 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.187381    2563 generic.go:356] "Generic (PLEG): container finished" podID="ede8eb3b-75ec-4141-91b6-c3ee3318efa2" containerID="06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210" exitCode=0
Sep 17 17:43:53.187840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.187428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bbb8964-tn2bn"
Sep 17 17:43:53.187840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.187464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bbb8964-tn2bn" event={"ID":"ede8eb3b-75ec-4141-91b6-c3ee3318efa2","Type":"ContainerDied","Data":"06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210"}
Sep 17 17:43:53.187840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.187516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bbb8964-tn2bn" event={"ID":"ede8eb3b-75ec-4141-91b6-c3ee3318efa2","Type":"ContainerDied","Data":"2680aa417f41d20672c2e29261532341c257b8d6b49e6520e2040d8a94f6ee19"}
Sep 17 17:43:53.187840 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.187537    2563 scope.go:117] "RemoveContainer" containerID="06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210"
Sep 17 17:43:53.196175 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.196157    2563 scope.go:117] "RemoveContainer" containerID="06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210"
Sep 17 17:43:53.196500 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:43:53.196478    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210\": container with ID starting with 06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210 not found: ID does not exist" containerID="06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210"
Sep 17 17:43:53.196546 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.196511    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210"} err="failed to get container status \"06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210\": rpc error: code = NotFound desc = could not find container \"06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210\": container with ID starting with 06a233861559828c3170f44e7dd90d26a7c464326f2fd0a26f628a992a7f8210 not found: ID does not exist"
Sep 17 17:43:53.205434 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.205411    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767bbb8964-tn2bn"]
Sep 17 17:43:53.207131 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:53.207111    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-767bbb8964-tn2bn"]
Sep 17 17:43:54.967591 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:43:54.967548    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede8eb3b-75ec-4141-91b6-c3ee3318efa2" path="/var/lib/kubelet/pods/ede8eb3b-75ec-4141-91b6-c3ee3318efa2/volumes"
Sep 17 17:44:00.739131 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.739091    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.832880 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.832835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:00.863397 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.863359    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-kz528"]
Sep 17 17:44:00.863805 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.863772    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede8eb3b-75ec-4141-91b6-c3ee3318efa2" containerName="console"
Sep 17 17:44:00.863913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.863807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede8eb3b-75ec-4141-91b6-c3ee3318efa2" containerName="console"
Sep 17 17:44:00.863969 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.863939    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede8eb3b-75ec-4141-91b6-c3ee3318efa2" containerName="console"
Sep 17 17:44:00.866873 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.866848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:00.869420 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.869398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 17:44:00.870152 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.870133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:44:00.870251 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.870213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-brhm5\""
Sep 17 17:44:00.870589 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.870571    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:44:00.878230 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.878207    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-kz528"]
Sep 17 17:44:00.916350 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.916313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-th4xh"]
Sep 17 17:44:00.918671 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.918653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:00.922072 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.922049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:44:00.922198 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.922049    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-m9dzw\""
Sep 17 17:44:00.932964 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.932936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-th4xh"]
Sep 17 17:44:00.992424 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.992334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7802228e-f1b0-4892-84fb-18dae4f656b1-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:00.992569 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:00.992461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ztf\" (UniqueName: \"kubernetes.io/projected/7802228e-f1b0-4892-84fb-18dae4f656b1-kube-api-access-r7ztf\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:01.085028 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.084985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.092989 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.092960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ztf\" (UniqueName: \"kubernetes.io/projected/7802228e-f1b0-4892-84fb-18dae4f656b1-kube-api-access-r7ztf\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:01.093164 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.093000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4e70ba67-b63a-42f4-b552-ee8926dadfb8-data\") pod \"seaweedfs-64568bcd49-th4xh\" (UID: \"4e70ba67-b63a-42f4-b552-ee8926dadfb8\") " pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.093164 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.093028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwsr\" (UniqueName: \"kubernetes.io/projected/4e70ba67-b63a-42f4-b552-ee8926dadfb8-kube-api-access-lnwsr\") pod \"seaweedfs-64568bcd49-th4xh\" (UID: \"4e70ba67-b63a-42f4-b552-ee8926dadfb8\") " pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.093164 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.093064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7802228e-f1b0-4892-84fb-18dae4f656b1-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:01.093344 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:44:01.093159    2563 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 17:44:01.093344 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:44:01.093234    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7802228e-f1b0-4892-84fb-18dae4f656b1-cert podName:7802228e-f1b0-4892-84fb-18dae4f656b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:44:01.593212671 +0000 UTC m=+527.120994489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7802228e-f1b0-4892-84fb-18dae4f656b1-cert") pod "llmisvc-controller-manager-645f9c8fd6-kz528" (UID: "7802228e-f1b0-4892-84fb-18dae4f656b1") : secret "llmisvc-webhook-server-cert" not found
Sep 17 17:44:01.102475 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.102446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ztf\" (UniqueName: \"kubernetes.io/projected/7802228e-f1b0-4892-84fb-18dae4f656b1-kube-api-access-r7ztf\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:01.179961 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.179927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.194012 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.193977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4e70ba67-b63a-42f4-b552-ee8926dadfb8-data\") pod \"seaweedfs-64568bcd49-th4xh\" (UID: \"4e70ba67-b63a-42f4-b552-ee8926dadfb8\") " pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.194154 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.194021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwsr\" (UniqueName: \"kubernetes.io/projected/4e70ba67-b63a-42f4-b552-ee8926dadfb8-kube-api-access-lnwsr\") pod \"seaweedfs-64568bcd49-th4xh\" (UID: \"4e70ba67-b63a-42f4-b552-ee8926dadfb8\") " pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.194437 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.194413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4e70ba67-b63a-42f4-b552-ee8926dadfb8-data\") pod \"seaweedfs-64568bcd49-th4xh\" (UID: \"4e70ba67-b63a-42f4-b552-ee8926dadfb8\") " pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.206425 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.206386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwsr\" (UniqueName: \"kubernetes.io/projected/4e70ba67-b63a-42f4-b552-ee8926dadfb8-kube-api-access-lnwsr\") pod \"seaweedfs-64568bcd49-th4xh\" (UID: \"4e70ba67-b63a-42f4-b552-ee8926dadfb8\") " pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.228140 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.228115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:01.347967 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.347921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.540368 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.540281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-th4xh"]
Sep 17 17:44:01.543213 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:44:01.543183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e70ba67_b63a_42f4_b552_ee8926dadfb8.slice/crio-6ddcd354c3870a048e01d57623afdae9c9791b6e2a2acbe955df5589179d5ff4 WatchSource:0}: Error finding container 6ddcd354c3870a048e01d57623afdae9c9791b6e2a2acbe955df5589179d5ff4: Status 404 returned error can't find the container with id 6ddcd354c3870a048e01d57623afdae9c9791b6e2a2acbe955df5589179d5ff4
Sep 17 17:44:01.598670 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.598639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7802228e-f1b0-4892-84fb-18dae4f656b1-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:01.601104 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.601079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7802228e-f1b0-4892-84fb-18dae4f656b1-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-kz528\" (UID: \"7802228e-f1b0-4892-84fb-18dae4f656b1\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:01.670023 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.669985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.768093 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.768060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:01.779038 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:01.779009    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:02.057260 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:02.057233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-kz528"]
Sep 17 17:44:02.083335 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:44:02.083292    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7802228e_f1b0_4892_84fb_18dae4f656b1.slice/crio-a441a73052554e3defdc1f9064cf12f714201ae64a883e61cd7d09aeef5a2a68 WatchSource:0}: Error finding container a441a73052554e3defdc1f9064cf12f714201ae64a883e61cd7d09aeef5a2a68: Status 404 returned error can't find the container with id a441a73052554e3defdc1f9064cf12f714201ae64a883e61cd7d09aeef5a2a68
Sep 17 17:44:02.231863 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:02.231823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-th4xh" event={"ID":"4e70ba67-b63a-42f4-b552-ee8926dadfb8","Type":"ContainerStarted","Data":"6ddcd354c3870a048e01d57623afdae9c9791b6e2a2acbe955df5589179d5ff4"}
Sep 17 17:44:02.233201 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:02.233175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528" event={"ID":"7802228e-f1b0-4892-84fb-18dae4f656b1","Type":"ContainerStarted","Data":"a441a73052554e3defdc1f9064cf12f714201ae64a883e61cd7d09aeef5a2a68"}
Sep 17 17:44:06.258036 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:06.257982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-th4xh" event={"ID":"4e70ba67-b63a-42f4-b552-ee8926dadfb8","Type":"ContainerStarted","Data":"4b3882bac6b7ae3327dab3ba7ece5a568c7c8a6d46f177e893eff28d68f5df1c"}
Sep 17 17:44:06.258457 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:06.258109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:06.259470 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:06.259445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528" event={"ID":"7802228e-f1b0-4892-84fb-18dae4f656b1","Type":"ContainerStarted","Data":"820d9cddca7820626955c5542201e0ffc727375e47d7ed74e5fbdb4e64e1ba35"}
Sep 17 17:44:06.259580 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:06.259494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:44:06.275978 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:06.275933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-th4xh" podStartSLOduration=2.306126104 podStartE2EDuration="6.275918654s" totalImagesPullingTime="3.96979255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:06.275298322 +0000 UTC m=+531.803080133" watchObservedRunningTime="2026-09-17 17:44:06.275918654 +0000 UTC m=+531.803700481"
Sep 17 17:44:06.295251 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:06.295205    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528" podStartSLOduration=2.900215331 podStartE2EDuration="6.295188887s" totalImagesPullingTime="3.394973556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:06.294433411 +0000 UTC m=+531.822215250" watchObservedRunningTime="2026-09-17 17:44:06.295188887 +0000 UTC m=+531.822970722"
Sep 17 17:44:09.145764 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:09.145681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:09.234320 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:09.234285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:12.267403 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:12.267373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-th4xh"
Sep 17 17:44:15.024384 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:15.024346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:29.754693 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:29.754652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:32.774233 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:32.774190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:37.267068 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:44:37.267029    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-kz528"
Sep 17 17:45:14.837682 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:14.837654    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:45:14.843828 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:14.843783    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:45:19.712965 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.712933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:19.773222 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.773184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-hmvmf"]
Sep 17 17:45:19.775592 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.775569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.778628 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.778604    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 17:45:19.778846 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.778826    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-q99fb\""
Sep 17 17:45:19.779146 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.779129    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 17:45:19.788690 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.788668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hmvmf"]
Sep 17 17:45:19.804720 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.804675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:19.879981 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.879946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb69v\" (UniqueName: \"kubernetes.io/projected/99467336-5c7e-4ff8-a159-4aa4d0b02150-kube-api-access-kb69v\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.880181 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.880001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-cert\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.880181 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.880107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-metrics-certs\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.910393 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.910358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:19.981146 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.981065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb69v\" (UniqueName: \"kubernetes.io/projected/99467336-5c7e-4ff8-a159-4aa4d0b02150-kube-api-access-kb69v\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.981146 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.981132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-cert\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.981343 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.981164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-metrics-certs\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.981343 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:45:19.981285    2563 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 17:45:19.981431 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:45:19.981366    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-cert podName:99467336-5c7e-4ff8-a159-4aa4d0b02150 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:20.481345422 +0000 UTC m=+606.009127249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-cert") pod "odh-model-controller-7b966d7984-hmvmf" (UID: "99467336-5c7e-4ff8-a159-4aa4d0b02150") : secret "odh-model-controller-webhook-cert" not found
Sep 17 17:45:19.983781 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.983755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-metrics-certs\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:19.992625 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:19.992596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb69v\" (UniqueName: \"kubernetes.io/projected/99467336-5c7e-4ff8-a159-4aa4d0b02150-kube-api-access-kb69v\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:20.045313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:20.045276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:20.148835 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:20.148772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:20.485557 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:20.485523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-cert\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:20.487983 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:20.487953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99467336-5c7e-4ff8-a159-4aa4d0b02150-cert\") pod \"odh-model-controller-7b966d7984-hmvmf\" (UID: \"99467336-5c7e-4ff8-a159-4aa4d0b02150\") " pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:20.686925 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:20.686876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:20.960723 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:20.960695    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hmvmf"]
Sep 17 17:45:20.964398 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:45:20.964375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99467336_5c7e_4ff8_a159_4aa4d0b02150.slice/crio-762bca0143cd6a538a431f64340a331bce6990022f21b9c8c865325951ddace5 WatchSource:0}: Error finding container 762bca0143cd6a538a431f64340a331bce6990022f21b9c8c865325951ddace5: Status 404 returned error can't find the container with id 762bca0143cd6a538a431f64340a331bce6990022f21b9c8c865325951ddace5
Sep 17 17:45:21.124928 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:21.124828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:21.224363 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:21.224333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:21.608071 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:21.608030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hmvmf" event={"ID":"99467336-5c7e-4ff8-a159-4aa4d0b02150","Type":"ContainerStarted","Data":"762bca0143cd6a538a431f64340a331bce6990022f21b9c8c865325951ddace5"}
Sep 17 17:45:23.619988 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:23.619949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hmvmf" event={"ID":"99467336-5c7e-4ff8-a159-4aa4d0b02150","Type":"ContainerStarted","Data":"c3f9addfd427da811b899186d4f93f0316854281341374eff48c3c4fbc056d00"}
Sep 17 17:45:23.620359 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:23.620026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:23.639060 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:23.638986    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-hmvmf" podStartSLOduration=2.119659262 podStartE2EDuration="4.63897011s" totalImagesPullingTime="2.519310848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:23.63839149 +0000 UTC m=+609.166173318" watchObservedRunningTime="2026-09-17 17:45:23.63897011 +0000 UTC m=+609.166751937"
Sep 17 17:45:34.627372 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:34.627335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-hmvmf"
Sep 17 17:45:57.938119 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:45:57.938083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:34.225872 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:46:34.225834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:55.471806 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:46:55.471749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:02.830287 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.830253    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-2ldws"]
Sep 17 17:47:02.833126 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.833110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.835713 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.835682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 17:47:02.835885 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.835868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 17:47:02.842021 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.842000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-2ldws"]
Sep 17 17:47:02.852160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.851943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/a30e69a3-1217-4da7-8eab-59cdf49c59d0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.852160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.851998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a30e69a3-1217-4da7-8eab-59cdf49c59d0-data\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.852160 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.852085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szhh\" (UniqueName: \"kubernetes.io/projected/a30e69a3-1217-4da7-8eab-59cdf49c59d0-kube-api-access-9szhh\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.953051 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.953008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a30e69a3-1217-4da7-8eab-59cdf49c59d0-data\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.953236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.953110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szhh\" (UniqueName: \"kubernetes.io/projected/a30e69a3-1217-4da7-8eab-59cdf49c59d0-kube-api-access-9szhh\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.953236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.953188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/a30e69a3-1217-4da7-8eab-59cdf49c59d0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.953467 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.953447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a30e69a3-1217-4da7-8eab-59cdf49c59d0-data\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.955643 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.955621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/a30e69a3-1217-4da7-8eab-59cdf49c59d0-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:02.963396 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:02.963375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szhh\" (UniqueName: \"kubernetes.io/projected/a30e69a3-1217-4da7-8eab-59cdf49c59d0-kube-api-access-9szhh\") pod \"seaweedfs-tls-custom-8b44d767d-2ldws\" (UID: \"a30e69a3-1217-4da7-8eab-59cdf49c59d0\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:03.143890 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:03.143750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws"
Sep 17 17:47:03.417691 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:03.417658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-2ldws"]
Sep 17 17:47:03.419816 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:47:03.419771    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30e69a3_1217_4da7_8eab_59cdf49c59d0.slice/crio-7d9eb33f45f25f98bec85c29056f7195705b7df7897c69b2098f76af9db47e38 WatchSource:0}: Error finding container 7d9eb33f45f25f98bec85c29056f7195705b7df7897c69b2098f76af9db47e38: Status 404 returned error can't find the container with id 7d9eb33f45f25f98bec85c29056f7195705b7df7897c69b2098f76af9db47e38
Sep 17 17:47:04.081923 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:04.081821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws" event={"ID":"a30e69a3-1217-4da7-8eab-59cdf49c59d0","Type":"ContainerStarted","Data":"8cc215e7c59bb18a13645437aa926272a466930d402e1af535378b9ebcdd6480"}
Sep 17 17:47:04.081923 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:04.081878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws" event={"ID":"a30e69a3-1217-4da7-8eab-59cdf49c59d0","Type":"ContainerStarted","Data":"7d9eb33f45f25f98bec85c29056f7195705b7df7897c69b2098f76af9db47e38"}
Sep 17 17:47:04.099395 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:04.099340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-2ldws" podStartSLOduration=1.828599332 podStartE2EDuration="2.099322424s" totalImagesPullingTime="270.723092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:04.098692878 +0000 UTC m=+709.626474704" watchObservedRunningTime="2026-09-17 17:47:04.099322424 +0000 UTC m=+709.627104249"
Sep 17 17:47:16.430501 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:16.430413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:16.965387 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:16.965349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:31.277373 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:31.277338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:31.378856 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:31.378818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:42.671763 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.671719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:42.775486 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.775450    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"]
Sep 17 17:47:42.779232 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.779214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:42.782261 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6eef31-predictor-serving-cert\""
Sep 17 17:47:42.782261 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782202    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:47:42.782261 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782200    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6eef31-kube-rbac-proxy-sar-config\""
Sep 17 17:47:42.782547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782205    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-6eef31\""
Sep 17 17:47:42.782547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782195    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:47:42.782547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-6eef31-dockercfg-tqhw6\""
Sep 17 17:47:42.782547 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.782229    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:47:42.797446 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.797418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"]
Sep 17 17:47:42.899881 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.899847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:42.900025 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.899904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zbz\" (UniqueName: \"kubernetes.io/projected/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-api-access-65zbz\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:42.900025 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.899937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kserve-provision-location\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:42.900025 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.900008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e0d99ab-9412-4c36-b7aa-444a230bc58f-proxy-tls\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:42.900173 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:42.900028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-cabundle-cert\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.001348 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.001251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e0d99ab-9412-4c36-b7aa-444a230bc58f-proxy-tls\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.001348 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.001296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-cabundle-cert\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.001348 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.001335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.001729 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.001367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zbz\" (UniqueName: \"kubernetes.io/projected/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-api-access-65zbz\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.001729 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.001392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kserve-provision-location\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.001901 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.001832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kserve-provision-location\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.002131 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.002108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-cabundle-cert\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.002131 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.002124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.004098 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.004075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e0d99ab-9412-4c36-b7aa-444a230bc58f-proxy-tls\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.010217 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.010191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zbz\" (UniqueName: \"kubernetes.io/projected/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-api-access-65zbz\") pod \"isvc-init-fail-6eef31-predictor-7f87c854dc-2przb\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") " pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.089938 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.089895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:43.372051 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.372026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"]
Sep 17 17:47:43.374271 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:47:43.374242    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0d99ab_9412_4c36_b7aa_444a230bc58f.slice/crio-3ccf35b00f2bb18f8f1f6cb884c362ee8a1564cfd8fe94c4973b266e2c9c028f WatchSource:0}: Error finding container 3ccf35b00f2bb18f8f1f6cb884c362ee8a1564cfd8fe94c4973b266e2c9c028f: Status 404 returned error can't find the container with id 3ccf35b00f2bb18f8f1f6cb884c362ee8a1564cfd8fe94c4973b266e2c9c028f
Sep 17 17:47:43.376560 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.376541    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:43.985391 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:43.985354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:44.280229 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:44.280135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" event={"ID":"4e0d99ab-9412-4c36-b7aa-444a230bc58f","Type":"ContainerStarted","Data":"3ccf35b00f2bb18f8f1f6cb884c362ee8a1564cfd8fe94c4973b266e2c9c028f"}
Sep 17 17:47:47.298902 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:47.298861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" event={"ID":"4e0d99ab-9412-4c36-b7aa-444a230bc58f","Type":"ContainerStarted","Data":"1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255"}
Sep 17 17:47:51.319080 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:51.319052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6eef31-predictor-7f87c854dc-2przb_4e0d99ab-9412-4c36-b7aa-444a230bc58f/storage-initializer/0.log"
Sep 17 17:47:51.319525 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:51.319093    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerID="1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255" exitCode=1
Sep 17 17:47:51.319525 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:51.319164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" event={"ID":"4e0d99ab-9412-4c36-b7aa-444a230bc58f","Type":"ContainerDied","Data":"1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255"}
Sep 17 17:47:52.325590 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:52.325507    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6eef31-predictor-7f87c854dc-2przb_4e0d99ab-9412-4c36-b7aa-444a230bc58f/storage-initializer/0.log"
Sep 17 17:47:52.325987 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:52.325628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" event={"ID":"4e0d99ab-9412-4c36-b7aa-444a230bc58f","Type":"ContainerStarted","Data":"7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89"}
Sep 17 17:47:52.555090 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:52.555052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"]
Sep 17 17:47:52.576501 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:52.576416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:53.330360 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:53.330324    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer" containerID="cri-o://7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89" gracePeriod=30
Sep 17 17:47:56.286247 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.286217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6eef31-predictor-7f87c854dc-2przb_4e0d99ab-9412-4c36-b7aa-444a230bc58f/storage-initializer/1.log"
Sep 17 17:47:56.286651 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.286633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6eef31-predictor-7f87c854dc-2przb_4e0d99ab-9412-4c36-b7aa-444a230bc58f/storage-initializer/0.log"
Sep 17 17:47:56.289035 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.289018    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:56.346163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6eef31-predictor-7f87c854dc-2przb_4e0d99ab-9412-4c36-b7aa-444a230bc58f/storage-initializer/1.log"
Sep 17 17:47:56.346559 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346541    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6eef31-predictor-7f87c854dc-2przb_4e0d99ab-9412-4c36-b7aa-444a230bc58f/storage-initializer/0.log"
Sep 17 17:47:56.346641 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346579    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerID="7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89" exitCode=1
Sep 17 17:47:56.346641 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" event={"ID":"4e0d99ab-9412-4c36-b7aa-444a230bc58f","Type":"ContainerDied","Data":"7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89"}
Sep 17 17:47:56.346753 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"
Sep 17 17:47:56.346753 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb" event={"ID":"4e0d99ab-9412-4c36-b7aa-444a230bc58f","Type":"ContainerDied","Data":"3ccf35b00f2bb18f8f1f6cb884c362ee8a1564cfd8fe94c4973b266e2c9c028f"}
Sep 17 17:47:56.346753 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.346706    2563 scope.go:117] "RemoveContainer" containerID="7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89"
Sep 17 17:47:56.355507 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.355488    2563 scope.go:117] "RemoveContainer" containerID="1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255"
Sep 17 17:47:56.363367 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.363349    2563 scope.go:117] "RemoveContainer" containerID="7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89"
Sep 17 17:47:56.363633 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:47:56.363617    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89\": container with ID starting with 7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89 not found: ID does not exist" containerID="7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89"
Sep 17 17:47:56.363683 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.363643    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89"} err="failed to get container status \"7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89\": rpc error: code = NotFound desc = could not find container \"7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89\": container with ID starting with 7cb7038380b4b6c4ed4368c676a3e323c5f7ffbb7cf216d72db823471670cb89 not found: ID does not exist"
Sep 17 17:47:56.363683 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.363662    2563 scope.go:117] "RemoveContainer" containerID="1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255"
Sep 17 17:47:56.363913 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:47:56.363898    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255\": container with ID starting with 1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255 not found: ID does not exist" containerID="1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255"
Sep 17 17:47:56.363963 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.363918    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255"} err="failed to get container status \"1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255\": rpc error: code = NotFound desc = could not find container \"1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255\": container with ID starting with 1c13017571bc8e53b4128fae3da21a27695fb6bd59adade11a7e67ad89058255 not found: ID does not exist"
Sep 17 17:47:56.433242 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-rbac-proxy-sar-config\") pod \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") "
Sep 17 17:47:56.433415 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e0d99ab-9412-4c36-b7aa-444a230bc58f-proxy-tls\") pod \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") "
Sep 17 17:47:56.433415 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kserve-provision-location\") pod \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") "
Sep 17 17:47:56.433501 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65zbz\" (UniqueName: \"kubernetes.io/projected/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-api-access-65zbz\") pod \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") "
Sep 17 17:47:56.433549 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-cabundle-cert\") pod \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\" (UID: \"4e0d99ab-9412-4c36-b7aa-444a230bc58f\") "
Sep 17 17:47:56.433602 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433560    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e0d99ab-9412-4c36-b7aa-444a230bc58f" (UID: "4e0d99ab-9412-4c36-b7aa-444a230bc58f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:56.433602 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433592    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e0d99ab-9412-4c36-b7aa-444a230bc58f" (UID: "4e0d99ab-9412-4c36-b7aa-444a230bc58f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:56.433839 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433783    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:47:56.433839 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433832    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:47:56.434012 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.433873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e0d99ab-9412-4c36-b7aa-444a230bc58f" (UID: "4e0d99ab-9412-4c36-b7aa-444a230bc58f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:47:56.435447 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.435421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0d99ab-9412-4c36-b7aa-444a230bc58f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e0d99ab-9412-4c36-b7aa-444a230bc58f" (UID: "4e0d99ab-9412-4c36-b7aa-444a230bc58f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:47:56.435615 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.435595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-api-access-65zbz" (OuterVolumeSpecName: "kube-api-access-65zbz") pod "4e0d99ab-9412-4c36-b7aa-444a230bc58f" (UID: "4e0d99ab-9412-4c36-b7aa-444a230bc58f"). InnerVolumeSpecName "kube-api-access-65zbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:56.535056 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.535005    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zbz\" (UniqueName: \"kubernetes.io/projected/4e0d99ab-9412-4c36-b7aa-444a230bc58f-kube-api-access-65zbz\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:47:56.535056 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.535051    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e0d99ab-9412-4c36-b7aa-444a230bc58f-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:47:56.535056 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.535061    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e0d99ab-9412-4c36-b7aa-444a230bc58f-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:47:56.690236 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.690200    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"]
Sep 17 17:47:56.696854 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.696820    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6eef31-predictor-7f87c854dc-2przb"]
Sep 17 17:47:56.959261 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:56.959216    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" path="/var/lib/kubelet/pods/4e0d99ab-9412-4c36-b7aa-444a230bc58f/volumes"
Sep 17 17:47:57.745852 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.745814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:57.782317 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"]
Sep 17 17:47:57.782668 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782652    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer"
Sep 17 17:47:57.782668 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782667    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer"
Sep 17 17:47:57.782863 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782778    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer"
Sep 17 17:47:57.782863 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782823    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer"
Sep 17 17:47:57.782955 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782913    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer"
Sep 17 17:47:57.782955 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.782923    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0d99ab-9412-4c36-b7aa-444a230bc58f" containerName="storage-initializer"
Sep 17 17:47:57.787613 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.787595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:57.790007 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.789980    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 17:47:57.790124 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.790039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:47:57.790186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.790119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 17:47:57.790186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.789992    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:47:57.790186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.790133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:47:57.790186 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.790129    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 17:47:57.790385 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.790272    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:47:57.796853 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.796831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"]
Sep 17 17:47:57.947077 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.947035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6sq\" (UniqueName: \"kubernetes.io/projected/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-api-access-qc6sq\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:57.947077 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.947081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:57.947366 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.947103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:57.947366 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.947246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:57.947366 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:57.947331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.048262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.048171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6sq\" (UniqueName: \"kubernetes.io/projected/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-api-access-qc6sq\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.048262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.048211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.048262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.048232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.048543 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.048273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.048543 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.048308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.048711 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:47:58.048594    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 17 17:47:58.048711 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:47:58.048664    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls podName:a1356176-2f4d-4a39-ac88-bf9ea592a5f6 nodeName:}" failed. No retries permitted until 2026-09-17 17:47:58.548642085 +0000 UTC m=+764.076423891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" (UID: "a1356176-2f4d-4a39-ac88-bf9ea592a5f6") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 17 17:47:58.049021 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.048997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.049138 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.049117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.049138 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.049131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.059597 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.059569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6sq\" (UniqueName: \"kubernetes.io/projected/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-api-access-qc6sq\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.242814 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.242759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:58.553491 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.553456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.555908 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.555878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-4vp4p\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.699048 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.699003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:47:58.979200 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:58.979172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"]
Sep 17 17:47:58.981264 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:47:58.981236    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1356176_2f4d_4a39_ac88_bf9ea592a5f6.slice/crio-26dc866d558403f7ba3ebfc044ae4f67660c7cedb8fc82b8db62502e76a0065e WatchSource:0}: Error finding container 26dc866d558403f7ba3ebfc044ae4f67660c7cedb8fc82b8db62502e76a0065e: Status 404 returned error can't find the container with id 26dc866d558403f7ba3ebfc044ae4f67660c7cedb8fc82b8db62502e76a0065e
Sep 17 17:47:59.363606 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:59.363509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerStarted","Data":"fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df"}
Sep 17 17:47:59.363606 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:47:59.363586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerStarted","Data":"26dc866d558403f7ba3ebfc044ae4f67660c7cedb8fc82b8db62502e76a0065e"}
Sep 17 17:48:03.384709 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:03.384674    2563 generic.go:356] "Generic (PLEG): container finished" podID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerID="fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df" exitCode=0
Sep 17 17:48:03.385163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:03.384751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerDied","Data":"fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df"}
Sep 17 17:48:26.532425 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:26.532387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerStarted","Data":"5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff"}
Sep 17 17:48:29.549868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:29.549832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerStarted","Data":"e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6"}
Sep 17 17:48:29.550262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:29.549964    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:48:29.572333 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:29.572273    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podStartSLOduration=3.85622209 podStartE2EDuration="32.572252532s" totalImagesPullingTime="25.091014577s" totalInitContainerRuntime="3.625015865s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:29.571691304 +0000 UTC m=+795.099473132" watchObservedRunningTime="2026-09-17 17:48:29.572252532 +0000 UTC m=+795.100034362"
Sep 17 17:48:30.554644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:30.554604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:48:30.556137 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:30.556106    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:48:31.559809 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:31.559766    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:48:36.564758 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:36.564726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:48:36.565265 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:36.565237    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:48:46.565995 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:46.565908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:48:56.565498 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:48:56.565453    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:49:06.565281 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:06.565236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:49:16.565962 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:16.565911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:49:26.566101 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:26.566053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:49:34.284770 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:34.284735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:36.565802 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:36.565751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:49:37.791106 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:37.791068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:37.891279 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:37.891243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:38.225551 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:38.225513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:46.565973 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:46.565937    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:49:52.396188 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:52.396101    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"]
Sep 17 17:49:52.396672 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:52.396391    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" containerID="cri-o://5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff" gracePeriod=30
Sep 17 17:49:52.396672 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:52.396421    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kube-rbac-proxy" containerID="cri-o://e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6" gracePeriod=30
Sep 17 17:49:52.420607 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:52.420577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:52.951413 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:52.951377    2563 generic.go:356] "Generic (PLEG): container finished" podID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerID="e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6" exitCode=2
Sep 17 17:49:52.951594 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:52.951444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerDied","Data":"e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6"}
Sep 17 17:49:56.560507 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:56.560462    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 17:49:56.565886 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:56.565857    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 17:49:57.516651 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.516606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:57.548558 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.548529    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"]
Sep 17 17:49:57.552497 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.552476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.555859 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.555838    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 17:49:57.555979 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.555857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 17:49:57.569710 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.569677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"]
Sep 17 17:49:57.638623 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.638585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt794\" (UniqueName: \"kubernetes.io/projected/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-api-access-mt794\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.638829 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.638638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.638829 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.638663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e514d5-bf62-4234-875d-8223cabf13d0-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.638829 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.638731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.638829 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.638808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.740257 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.740222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt794\" (UniqueName: \"kubernetes.io/projected/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-api-access-mt794\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.740445 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.740278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.740445 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.740314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e514d5-bf62-4234-875d-8223cabf13d0-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.740445 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.740342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.740445 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.740401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.741025 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:49:57.740706    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 17:49:57.741025 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:49:57.740773    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls podName:e1e514d5-bf62-4234-875d-8223cabf13d0 nodeName:}" failed. No retries permitted until 2026-09-17 17:49:58.240754483 +0000 UTC m=+883.768536301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-lff7m" (UID: "e1e514d5-bf62-4234-875d-8223cabf13d0") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 17:49:57.741214 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.741075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e514d5-bf62-4234-875d-8223cabf13d0-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.741214 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.741139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.741214 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.741139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.751823 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.751775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt794\" (UniqueName: \"kubernetes.io/projected/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-api-access-mt794\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:57.857986 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.857962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:49:57.978760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.978714    2563 generic.go:356] "Generic (PLEG): container finished" podID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerID="5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff" exitCode=0
Sep 17 17:49:57.978987 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.978813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"
Sep 17 17:49:57.978987 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.978812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerDied","Data":"5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff"}
Sep 17 17:49:57.978987 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.978878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p" event={"ID":"a1356176-2f4d-4a39-ac88-bf9ea592a5f6","Type":"ContainerDied","Data":"26dc866d558403f7ba3ebfc044ae4f67660c7cedb8fc82b8db62502e76a0065e"}
Sep 17 17:49:57.978987 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.978909    2563 scope.go:117] "RemoveContainer" containerID="e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6"
Sep 17 17:49:57.987576 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.987558    2563 scope.go:117] "RemoveContainer" containerID="5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff"
Sep 17 17:49:57.995119 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:57.995100    2563 scope.go:117] "RemoveContainer" containerID="fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df"
Sep 17 17:49:58.003101 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.003080    2563 scope.go:117] "RemoveContainer" containerID="e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6"
Sep 17 17:49:58.003373 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:49:58.003353    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6\": container with ID starting with e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6 not found: ID does not exist" containerID="e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6"
Sep 17 17:49:58.003449 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.003388    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6"} err="failed to get container status \"e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6\": rpc error: code = NotFound desc = could not find container \"e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6\": container with ID starting with e6a39a64d5db235f65c93c577ef02b0473ac6f95cbd3bff07c56a847c3b36ba6 not found: ID does not exist"
Sep 17 17:49:58.003449 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.003416    2563 scope.go:117] "RemoveContainer" containerID="5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff"
Sep 17 17:49:58.003706 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:49:58.003690    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff\": container with ID starting with 5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff not found: ID does not exist" containerID="5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff"
Sep 17 17:49:58.003776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.003719    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff"} err="failed to get container status \"5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff\": rpc error: code = NotFound desc = could not find container \"5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff\": container with ID starting with 5dc8009cb411c25acc7b334ea7aee781e65ddc8cdf690ca3ff20aa5f89b23aff not found: ID does not exist"
Sep 17 17:49:58.003776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.003743    2563 scope.go:117] "RemoveContainer" containerID="fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df"
Sep 17 17:49:58.003989 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:49:58.003972    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df\": container with ID starting with fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df not found: ID does not exist" containerID="fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df"
Sep 17 17:49:58.004039 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.003996    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df"} err="failed to get container status \"fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df\": rpc error: code = NotFound desc = could not find container \"fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df\": container with ID starting with fa3feb9dcaaf6d1ddcb6089e4482822ac302cfd856e84bba39c6b79e0e7086df not found: ID does not exist"
Sep 17 17:49:58.042812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.042719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-cabundle-cert\") pod \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") "
Sep 17 17:49:58.042812 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.042776    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-rbac-proxy-sar-config\") pod \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") "
Sep 17 17:49:58.043053 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.042859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls\") pod \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") "
Sep 17 17:49:58.043053 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.042903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc6sq\" (UniqueName: \"kubernetes.io/projected/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-api-access-qc6sq\") pod \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") "
Sep 17 17:49:58.043053 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.042928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kserve-provision-location\") pod \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\" (UID: \"a1356176-2f4d-4a39-ac88-bf9ea592a5f6\") "
Sep 17 17:49:58.043260 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.043233    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1356176-2f4d-4a39-ac88-bf9ea592a5f6" (UID: "a1356176-2f4d-4a39-ac88-bf9ea592a5f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:58.043334 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.043283    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1356176-2f4d-4a39-ac88-bf9ea592a5f6" (UID: "a1356176-2f4d-4a39-ac88-bf9ea592a5f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:58.043390 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.043344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1356176-2f4d-4a39-ac88-bf9ea592a5f6" (UID: "a1356176-2f4d-4a39-ac88-bf9ea592a5f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:58.045099 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.045073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1356176-2f4d-4a39-ac88-bf9ea592a5f6" (UID: "a1356176-2f4d-4a39-ac88-bf9ea592a5f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:58.045182 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.045163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-api-access-qc6sq" (OuterVolumeSpecName: "kube-api-access-qc6sq") pod "a1356176-2f4d-4a39-ac88-bf9ea592a5f6" (UID: "a1356176-2f4d-4a39-ac88-bf9ea592a5f6"). InnerVolumeSpecName "kube-api-access-qc6sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:58.097320 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.097273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:58.144398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.144354    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:49:58.144398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.144391    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:49:58.144398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.144407    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:49:58.144639 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.144419    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6sq\" (UniqueName: \"kubernetes.io/projected/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kube-api-access-qc6sq\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:49:58.144639 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.144431    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1356176-2f4d-4a39-ac88-bf9ea592a5f6-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:49:58.245822 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.245760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:58.248176 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.248153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-lff7m\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:58.303545 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.303457    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"]
Sep 17 17:49:58.307264 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.307233    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-4vp4p"]
Sep 17 17:49:58.463319 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.463279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:49:58.738697 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.738660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"]
Sep 17 17:49:58.742523 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:49:58.742497    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e514d5_bf62_4234_875d_8223cabf13d0.slice/crio-773410946af2a812c8fec0fda09991483ef830746d492a036a646c0041a13015 WatchSource:0}: Error finding container 773410946af2a812c8fec0fda09991483ef830746d492a036a646c0041a13015: Status 404 returned error can't find the container with id 773410946af2a812c8fec0fda09991483ef830746d492a036a646c0041a13015
Sep 17 17:49:58.959109 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.959076    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" path="/var/lib/kubelet/pods/a1356176-2f4d-4a39-ac88-bf9ea592a5f6/volumes"
Sep 17 17:49:58.987110 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.987070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerStarted","Data":"1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212"}
Sep 17 17:49:58.987283 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:49:58.987153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerStarted","Data":"773410946af2a812c8fec0fda09991483ef830746d492a036a646c0041a13015"}
Sep 17 17:50:03.007703 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:03.007670    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerID="1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212" exitCode=0
Sep 17 17:50:03.008100 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:03.007742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerDied","Data":"1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212"}
Sep 17 17:50:03.140182 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:03.140116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:03.241625 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:03.241591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:04.015530 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:04.015496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerStarted","Data":"505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708"}
Sep 17 17:50:04.015935 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:04.015565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerStarted","Data":"642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39"}
Sep 17 17:50:04.015935 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:04.015765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:50:04.041653 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:04.041582    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podStartSLOduration=3.18093522 podStartE2EDuration="7.041557327s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.860622107s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:04.039312338 +0000 UTC m=+889.567094165" watchObservedRunningTime="2026-09-17 17:50:04.041557327 +0000 UTC m=+889.569339154"
Sep 17 17:50:05.020227 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:05.020191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:50:05.021595 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:05.021564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:50:06.024552 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:06.024509    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:50:11.029330 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:11.029252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:50:11.029829 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:11.029784    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:50:14.876683 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:14.876650    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:50:14.879934 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:14.879911    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:50:21.030625 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:21.030580    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:50:31.030047 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:31.030005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:50:41.030201 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:41.030159    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:50:51.029878 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:50:51.029831    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:51:01.030637 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:01.030593    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:51:11.030157 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:11.030115    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:51:12.942292 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:12.942255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:13.018479 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:13.018439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:13.742684 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:13.742648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:20.953699 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:20.953643    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:51:30.958709 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:30.958680    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:51:32.574468 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:32.574426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:32.675060 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:32.675028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:33.110913 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:33.110878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:33.752142 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:33.752105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:38.178760 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:38.178679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"]
Sep 17 17:51:38.179170 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:38.178969    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" containerID="cri-o://642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39" gracePeriod=30
Sep 17 17:51:38.179170 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:38.179005    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kube-rbac-proxy" containerID="cri-o://505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708" gracePeriod=30
Sep 17 17:51:38.201800 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:38.201752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:38.468132 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:38.468040    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerID="505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708" exitCode=2
Sep 17 17:51:38.468283 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:38.468114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerDied","Data":"505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708"}
Sep 17 17:51:40.954437 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:40.954389    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 17:51:41.025312 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:41.025258    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 17:51:43.368380 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368343    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"]
Sep 17 17:51:43.368765 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container"
Sep 17 17:51:43.368765 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368753    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container"
Sep 17 17:51:43.368765 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368765    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="storage-initializer"
Sep 17 17:51:43.368897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368771    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="storage-initializer"
Sep 17 17:51:43.368897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368783    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kube-rbac-proxy"
Sep 17 17:51:43.368897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368801    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kube-rbac-proxy"
Sep 17 17:51:43.368897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368867    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kserve-container"
Sep 17 17:51:43.368897 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.368878    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1356176-2f4d-4a39-ac88-bf9ea592a5f6" containerName="kube-rbac-proxy"
Sep 17 17:51:43.371228 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.371188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:43.372111 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.372094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.374362 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.374341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 17:51:43.374478 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.374342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 17:51:43.384177 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.384147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"]
Sep 17 17:51:43.463156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.463122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.463313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.463167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.463313 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.463187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.463392 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.463319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2c4\" (UniqueName: \"kubernetes.io/projected/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-api-access-zf2c4\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.463392 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.463355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.564361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2c4\" (UniqueName: \"kubernetes.io/projected/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-api-access-zf2c4\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564398 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.564400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.564465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.564493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564644 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.564521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564968 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:51:43.564912    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 17:51:43.564968 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.564962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.564968 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:51:43.564988    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls podName:e6fc7876-7bf4-4b7d-be4b-794d0df42d7d nodeName:}" failed. No retries permitted until 2026-09-17 17:51:44.064966559 +0000 UTC m=+989.592748380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" (UID: "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 17 17:51:43.565332 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.565314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.565416 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.565394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.578807 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.578763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2c4\" (UniqueName: \"kubernetes.io/projected/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-api-access-zf2c4\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:43.634201 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.634135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:51:43.766917 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.766877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls\") pod \"e1e514d5-bf62-4234-875d-8223cabf13d0\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") "
Sep 17 17:51:43.767130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.766932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e514d5-bf62-4234-875d-8223cabf13d0-kserve-provision-location\") pod \"e1e514d5-bf62-4234-875d-8223cabf13d0\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") "
Sep 17 17:51:43.767130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.766954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-cabundle-cert\") pod \"e1e514d5-bf62-4234-875d-8223cabf13d0\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") "
Sep 17 17:51:43.767130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.766973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt794\" (UniqueName: \"kubernetes.io/projected/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-api-access-mt794\") pod \"e1e514d5-bf62-4234-875d-8223cabf13d0\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") "
Sep 17 17:51:43.767130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.767011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-rbac-proxy-sar-config\") pod \"e1e514d5-bf62-4234-875d-8223cabf13d0\" (UID: \"e1e514d5-bf62-4234-875d-8223cabf13d0\") "
Sep 17 17:51:43.767405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.767376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e514d5-bf62-4234-875d-8223cabf13d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1e514d5-bf62-4234-875d-8223cabf13d0" (UID: "e1e514d5-bf62-4234-875d-8223cabf13d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:43.767459 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.767433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1e514d5-bf62-4234-875d-8223cabf13d0" (UID: "e1e514d5-bf62-4234-875d-8223cabf13d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:43.767500 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.767469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1e514d5-bf62-4234-875d-8223cabf13d0" (UID: "e1e514d5-bf62-4234-875d-8223cabf13d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:43.769198 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.769169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1e514d5-bf62-4234-875d-8223cabf13d0" (UID: "e1e514d5-bf62-4234-875d-8223cabf13d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:43.769296 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.769192    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-api-access-mt794" (OuterVolumeSpecName: "kube-api-access-mt794") pod "e1e514d5-bf62-4234-875d-8223cabf13d0" (UID: "e1e514d5-bf62-4234-875d-8223cabf13d0"). InnerVolumeSpecName "kube-api-access-mt794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:43.868703 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.868667    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e514d5-bf62-4234-875d-8223cabf13d0-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.868703 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.868700    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e514d5-bf62-4234-875d-8223cabf13d0-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.868703 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.868711    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.868964 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.868720    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt794\" (UniqueName: \"kubernetes.io/projected/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-api-access-mt794\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.868964 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:43.868729    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e514d5-bf62-4234-875d-8223cabf13d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:51:44.048845 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.048810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:44.070654 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.070613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:44.073172 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.073149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:44.283851 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.283812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:44.497197 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.497162    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerID="642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39" exitCode=0
Sep 17 17:51:44.497678 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.497227    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"
Sep 17 17:51:44.497678 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.497230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerDied","Data":"642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39"}
Sep 17 17:51:44.497678 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.497378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m" event={"ID":"e1e514d5-bf62-4234-875d-8223cabf13d0","Type":"ContainerDied","Data":"773410946af2a812c8fec0fda09991483ef830746d492a036a646c0041a13015"}
Sep 17 17:51:44.497678 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.497409    2563 scope.go:117] "RemoveContainer" containerID="505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708"
Sep 17 17:51:44.506686 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.506666    2563 scope.go:117] "RemoveContainer" containerID="642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39"
Sep 17 17:51:44.515100 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.515079    2563 scope.go:117] "RemoveContainer" containerID="1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212"
Sep 17 17:51:44.522214 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.522187    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"]
Sep 17 17:51:44.525833 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.525787    2563 scope.go:117] "RemoveContainer" containerID="505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708"
Sep 17 17:51:44.526477 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:51:44.526447    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708\": container with ID starting with 505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708 not found: ID does not exist" containerID="505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708"
Sep 17 17:51:44.526573 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.526484    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708"} err="failed to get container status \"505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708\": rpc error: code = NotFound desc = could not find container \"505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708\": container with ID starting with 505496652dcadf829493a8b54beb85643041c09321aa3b9c59135faa3a730708 not found: ID does not exist"
Sep 17 17:51:44.526573 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.526507    2563 scope.go:117] "RemoveContainer" containerID="642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39"
Sep 17 17:51:44.526861 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:51:44.526836    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39\": container with ID starting with 642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39 not found: ID does not exist" containerID="642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39"
Sep 17 17:51:44.526968 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.526867    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39"} err="failed to get container status \"642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39\": rpc error: code = NotFound desc = could not find container \"642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39\": container with ID starting with 642bc88cea06554ef23f034bfca16adc094df686ed2ec92d815c7e563d4fcf39 not found: ID does not exist"
Sep 17 17:51:44.526968 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.526890    2563 scope.go:117] "RemoveContainer" containerID="1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212"
Sep 17 17:51:44.527244 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:51:44.527213    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212\": container with ID starting with 1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212 not found: ID does not exist" containerID="1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212"
Sep 17 17:51:44.527297 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.527251    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212"} err="failed to get container status \"1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212\": rpc error: code = NotFound desc = could not find container \"1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212\": container with ID starting with 1fc637a6850d61f18c3b0e246d8dfbc4cf954be44fc7fbde39474085359f5212 not found: ID does not exist"
Sep 17 17:51:44.528363 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.528341    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-lff7m"]
Sep 17 17:51:44.559622 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.559595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"]
Sep 17 17:51:44.562297 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:51:44.562263    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fc7876_7bf4_4b7d_be4b_794d0df42d7d.slice/crio-2f65548504917e86071e8b1392ba9ab8266200dae95214cdcf97c0153ae810a9 WatchSource:0}: Error finding container 2f65548504917e86071e8b1392ba9ab8266200dae95214cdcf97c0153ae810a9: Status 404 returned error can't find the container with id 2f65548504917e86071e8b1392ba9ab8266200dae95214cdcf97c0153ae810a9
Sep 17 17:51:44.964353 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:44.964312    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" path="/var/lib/kubelet/pods/e1e514d5-bf62-4234-875d-8223cabf13d0/volumes"
Sep 17 17:51:45.503894 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:45.503852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerStarted","Data":"b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c"}
Sep 17 17:51:45.504410 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:45.503927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerStarted","Data":"2f65548504917e86071e8b1392ba9ab8266200dae95214cdcf97c0153ae810a9"}
Sep 17 17:51:48.520511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:48.520477    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerID="b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c" exitCode=0
Sep 17 17:51:48.520932 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:48.520550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerDied","Data":"b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c"}
Sep 17 17:51:49.527427 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:49.527397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerStarted","Data":"66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc"}
Sep 17 17:51:49.527868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:49.527457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerStarted","Data":"12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183"}
Sep 17 17:51:49.527868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:49.527652    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:49.551269 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:49.551209    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podStartSLOduration=2.705748197 podStartE2EDuration="6.551194155s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845445958s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:49.549993559 +0000 UTC m=+995.077775396" watchObservedRunningTime="2026-09-17 17:51:49.551194155 +0000 UTC m=+995.078975978"
Sep 17 17:51:50.537244 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:50.537208    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:50.539143 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:50.539102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:51:51.542409 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:51.542366    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:51:56.547553 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:56.547523    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:51:56.548034 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:51:56.548005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:52:06.548492 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:52:06.548446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:52:16.548639 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:52:16.548603    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:52:26.548893 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:52:26.548853    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:52:36.548340 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:52:36.548298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:52:46.548498 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:52:46.548454    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:52:56.548936 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:52:56.548896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:53:06.549527 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:06.549488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:53:08.485134 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.485054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:08.656266 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656231    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"]
Sep 17 17:53:08.656685 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656667    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container"
Sep 17 17:53:08.656776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656688    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container"
Sep 17 17:53:08.656776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656718    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kube-rbac-proxy"
Sep 17 17:53:08.656776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656728    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kube-rbac-proxy"
Sep 17 17:53:08.656776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656747    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="storage-initializer"
Sep 17 17:53:08.656776 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656757    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="storage-initializer"
Sep 17 17:53:08.657055 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656885    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kserve-container"
Sep 17 17:53:08.657055 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.656898    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e514d5-bf62-4234-875d-8223cabf13d0" containerName="kube-rbac-proxy"
Sep 17 17:53:08.660623 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.660600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.665761 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.665731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 17:53:08.665917 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.665744    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 17:53:08.678774 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.678739    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"]
Sep 17 17:53:08.822509 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.822419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.822509 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.822457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.822740 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.822541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.822740 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.822607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.822740 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.822638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkh5l\" (UniqueName: \"kubernetes.io/projected/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-api-access-vkh5l\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.923570 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.923535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.923761 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.923582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.923761 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.923698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkh5l\" (UniqueName: \"kubernetes.io/projected/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-api-access-vkh5l\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.923896 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.923874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.923941 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.923911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.924262 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.924239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.924376 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.924264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.924613 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.924591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.926190 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.926170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.935039 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.935008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkh5l\" (UniqueName: \"kubernetes.io/projected/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-api-access-vkh5l\") pod \"isvc-autogluon-v1-predictor-574b5b7495-b59l8\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:08.971987 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:08.971950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:53:09.255219 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:09.255189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"]
Sep 17 17:53:09.257611 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:53:09.257585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2939886_c19b_4c70_ba4b_4afb2bbcff9c.slice/crio-a472ab878fdecab897fb5a7373861fc155c61d427ef162ca3d93ac3e25797eed WatchSource:0}: Error finding container a472ab878fdecab897fb5a7373861fc155c61d427ef162ca3d93ac3e25797eed: Status 404 returned error can't find the container with id a472ab878fdecab897fb5a7373861fc155c61d427ef162ca3d93ac3e25797eed
Sep 17 17:53:09.737170 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:09.737136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:09.907973 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:09.907934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerStarted","Data":"6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9"}
Sep 17 17:53:09.908168 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:09.908009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerStarted","Data":"a472ab878fdecab897fb5a7373861fc155c61d427ef162ca3d93ac3e25797eed"}
Sep 17 17:53:13.892881 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:13.892846    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"]
Sep 17 17:53:13.893297 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:13.893084    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" containerID="cri-o://12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183" gracePeriod=30
Sep 17 17:53:13.893297 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:13.893133    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kube-rbac-proxy" containerID="cri-o://66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc" gracePeriod=30
Sep 17 17:53:13.919905 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:13.919872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:14.938021 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:14.937983    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerID="66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc" exitCode=2
Sep 17 17:53:14.938431 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:14.938056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerDied","Data":"66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc"}
Sep 17 17:53:16.542889 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:16.542837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 17:53:16.548203 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:16.548175    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 17:53:19.051722 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.051688    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:19.420301 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.420216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:53:19.740344 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.740317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:53:19.928448 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.928415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kserve-provision-location\") pod \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") "
Sep 17 17:53:19.928624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.928505    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls\") pod \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") "
Sep 17 17:53:19.928624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.928556    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf2c4\" (UniqueName: \"kubernetes.io/projected/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-api-access-zf2c4\") pod \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") "
Sep 17 17:53:19.928713 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.928684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-rbac-proxy-sar-config\") pod \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") "
Sep 17 17:53:19.928755 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.928726    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-cabundle-cert\") pod \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\" (UID: \"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d\") "
Sep 17 17:53:19.928873 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.928850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" (UID: "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:19.929114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.929094    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:53:19.929114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.929104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" (UID: "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:19.929271 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.929100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" (UID: "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:19.930687 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.930669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-api-access-zf2c4" (OuterVolumeSpecName: "kube-api-access-zf2c4") pod "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" (UID: "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d"). InnerVolumeSpecName "kube-api-access-zf2c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:19.930687 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.930675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" (UID: "e6fc7876-7bf4-4b7d-be4b-794d0df42d7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:19.965614 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.965576    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerID="12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183" exitCode=0
Sep 17 17:53:19.965770 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.965647    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"
Sep 17 17:53:19.965770 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.965666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerDied","Data":"12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183"}
Sep 17 17:53:19.965770 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.965729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg" event={"ID":"e6fc7876-7bf4-4b7d-be4b-794d0df42d7d","Type":"ContainerDied","Data":"2f65548504917e86071e8b1392ba9ab8266200dae95214cdcf97c0153ae810a9"}
Sep 17 17:53:19.965770 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.965758    2563 scope.go:117] "RemoveContainer" containerID="66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc"
Sep 17 17:53:19.974675 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.974659    2563 scope.go:117] "RemoveContainer" containerID="12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183"
Sep 17 17:53:19.982955 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.982934    2563 scope.go:117] "RemoveContainer" containerID="b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c"
Sep 17 17:53:19.990899 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.990876    2563 scope.go:117] "RemoveContainer" containerID="66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc"
Sep 17 17:53:19.991291 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:53:19.991260    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc\": container with ID starting with 66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc not found: ID does not exist" containerID="66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc"
Sep 17 17:53:19.991454 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.991299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc"} err="failed to get container status \"66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc\": rpc error: code = NotFound desc = could not find container \"66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc\": container with ID starting with 66f71450a150617b870dc36eae0b5c5c2d23ab908ff02517234a6b8145fc2cbc not found: ID does not exist"
Sep 17 17:53:19.991454 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.991324    2563 scope.go:117] "RemoveContainer" containerID="12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183"
Sep 17 17:53:19.991818 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:53:19.991776    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183\": container with ID starting with 12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183 not found: ID does not exist" containerID="12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183"
Sep 17 17:53:19.991914 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.991826    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183"} err="failed to get container status \"12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183\": rpc error: code = NotFound desc = could not find container \"12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183\": container with ID starting with 12605d059c56d99dc15cea6885f6f864bd7d4281660c7ee4bddcd04782926183 not found: ID does not exist"
Sep 17 17:53:19.991914 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.991848    2563 scope.go:117] "RemoveContainer" containerID="b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c"
Sep 17 17:53:19.992145 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:53:19.992123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c\": container with ID starting with b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c not found: ID does not exist" containerID="b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c"
Sep 17 17:53:19.992190 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.992152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c"} err="failed to get container status \"b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c\": rpc error: code = NotFound desc = could not find container \"b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c\": container with ID starting with b08f4c1f6d1d7a571d7729a6e9a5c77b7db93dc3c62e3a1c9acf528c8fb7ae0c not found: ID does not exist"
Sep 17 17:53:19.994428 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.994407    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"]
Sep 17 17:53:19.996018 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:19.995998    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-9vnxg"]
Sep 17 17:53:20.030002 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:20.029961    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:53:20.030002 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:20.029995    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2c4\" (UniqueName: \"kubernetes.io/projected/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-api-access-zf2c4\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:53:20.030002 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:20.030006    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:53:20.030002 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:20.030016    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:53:20.958801 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:20.958753    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" path="/var/lib/kubelet/pods/e6fc7876-7bf4-4b7d-be4b-794d0df42d7d/volumes"
Sep 17 17:53:40.067150 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:40.067113    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerID="6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9" exitCode=0
Sep 17 17:53:40.067668 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:40.067170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerDied","Data":"6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9"}
Sep 17 17:53:40.068594 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:53:40.068571    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:54:58.444759 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:54:58.444678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:09.270210 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:09.270171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:14.354684 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:14.354644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:14.649381 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:14.649277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:33.202728 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:33.202695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:55:33.215038 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:33.202695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 17:55:34.688274 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:34.688230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerStarted","Data":"9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1"}
Sep 17 17:55:34.688810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:34.688296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerStarted","Data":"48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f"}
Sep 17 17:55:34.688810 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:34.688329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:55:34.710130 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:34.710069    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podStartSLOduration=3.211132125 podStartE2EDuration="2m26.71005214s" totalImagesPullingTime="1m53.723309825s" totalInitContainerRuntime="29.77561019s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:34.707658152 +0000 UTC m=+1220.235439983" watchObservedRunningTime="2026-09-17 17:55:34.71005214 +0000 UTC m=+1220.237833966"
Sep 17 17:55:35.693277 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:35.693235    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:55:35.694747 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:35.694717    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:55:36.699091 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:36.699047    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:55:41.703920 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:41.703891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:55:41.704391 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:41.704294    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 17:55:49.912645 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:49.912607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:51.705585 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:55:51.705557    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:56:05.225712 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:05.225679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"]
Sep 17 17:56:05.226229 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:05.225930    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container" containerID="cri-o://48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f" gracePeriod=30
Sep 17 17:56:05.226229 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:05.225958    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kube-rbac-proxy" containerID="cri-o://9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1" gracePeriod=30
Sep 17 17:56:05.253211 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:05.253180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:05.841744 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:05.841712    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerID="9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1" exitCode=2
Sep 17 17:56:05.841976 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:05.841785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerDied","Data":"9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1"}
Sep 17 17:56:06.700147 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.700108    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 17:56:06.802207 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.802179    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:56:06.849474 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.849436    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerID="48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f" exitCode=0
Sep 17 17:56:06.849645 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.849507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"
Sep 17 17:56:06.849645 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.849509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerDied","Data":"48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f"}
Sep 17 17:56:06.849645 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.849630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8" event={"ID":"a2939886-c19b-4c70-ba4b-4afb2bbcff9c","Type":"ContainerDied","Data":"a472ab878fdecab897fb5a7373861fc155c61d427ef162ca3d93ac3e25797eed"}
Sep 17 17:56:06.849769 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.849657    2563 scope.go:117] "RemoveContainer" containerID="9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1"
Sep 17 17:56:06.858037 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.858020    2563 scope.go:117] "RemoveContainer" containerID="48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f"
Sep 17 17:56:06.865615 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.865597    2563 scope.go:117] "RemoveContainer" containerID="6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9"
Sep 17 17:56:06.872956 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.872937    2563 scope.go:117] "RemoveContainer" containerID="9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1"
Sep 17 17:56:06.873266 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:56:06.873243    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1\": container with ID starting with 9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1 not found: ID does not exist" containerID="9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1"
Sep 17 17:56:06.873337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.873275    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1"} err="failed to get container status \"9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1\": rpc error: code = NotFound desc = could not find container \"9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1\": container with ID starting with 9f9522f5ddfa93e433383a6c96fe0ce0a3d7a500425a51903f20089e9fc037e1 not found: ID does not exist"
Sep 17 17:56:06.873337 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.873294    2563 scope.go:117] "RemoveContainer" containerID="48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f"
Sep 17 17:56:06.873586 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:56:06.873561    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f\": container with ID starting with 48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f not found: ID does not exist" containerID="48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f"
Sep 17 17:56:06.873648 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.873586    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f"} err="failed to get container status \"48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f\": rpc error: code = NotFound desc = could not find container \"48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f\": container with ID starting with 48c8f51f1d2c70232aff4bee4974158417889334856a362e1f3cbb685ea9f86f not found: ID does not exist"
Sep 17 17:56:06.873648 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.873602    2563 scope.go:117] "RemoveContainer" containerID="6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9"
Sep 17 17:56:06.873916 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:56:06.873892    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9\": container with ID starting with 6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9 not found: ID does not exist" containerID="6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9"
Sep 17 17:56:06.874032 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.873922    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9"} err="failed to get container status \"6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9\": rpc error: code = NotFound desc = could not find container \"6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9\": container with ID starting with 6480f171723e39234ee4b423f650f88ee0e4957cadf2e32c6bfb8c8424b3aeb9 not found: ID does not exist"
Sep 17 17:56:06.990922 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.990889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-rbac-proxy-sar-config\") pod \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") "
Sep 17 17:56:06.991064 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.990929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-proxy-tls\") pod \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") "
Sep 17 17:56:06.991064 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.990971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkh5l\" (UniqueName: \"kubernetes.io/projected/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-api-access-vkh5l\") pod \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") "
Sep 17 17:56:06.991064 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.991031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-cabundle-cert\") pod \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") "
Sep 17 17:56:06.991239 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.991095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kserve-provision-location\") pod \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\" (UID: \"a2939886-c19b-4c70-ba4b-4afb2bbcff9c\") "
Sep 17 17:56:06.991436 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.991409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2939886-c19b-4c70-ba4b-4afb2bbcff9c" (UID: "a2939886-c19b-4c70-ba4b-4afb2bbcff9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:06.991553 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.991533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2939886-c19b-4c70-ba4b-4afb2bbcff9c" (UID: "a2939886-c19b-4c70-ba4b-4afb2bbcff9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:06.993163 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.993137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-api-access-vkh5l" (OuterVolumeSpecName: "kube-api-access-vkh5l") pod "a2939886-c19b-4c70-ba4b-4afb2bbcff9c" (UID: "a2939886-c19b-4c70-ba4b-4afb2bbcff9c"). InnerVolumeSpecName "kube-api-access-vkh5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:06.993252 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.993186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2939886-c19b-4c70-ba4b-4afb2bbcff9c" (UID: "a2939886-c19b-4c70-ba4b-4afb2bbcff9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:06.994612 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:06.994588    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2939886-c19b-4c70-ba4b-4afb2bbcff9c" (UID: "a2939886-c19b-4c70-ba4b-4afb2bbcff9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.092766 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.092736    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.092766 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.092761    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.092766 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.092770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.093004 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.092779    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.093004 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.092787    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkh5l\" (UniqueName: \"kubernetes.io/projected/a2939886-c19b-4c70-ba4b-4afb2bbcff9c-kube-api-access-vkh5l\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.173922 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.173888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"]
Sep 17 17:56:07.179145 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:07.179115    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-b59l8"]
Sep 17 17:56:08.958936 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:08.958899    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" path="/var/lib/kubelet/pods/a2939886-c19b-4c70-ba4b-4afb2bbcff9c/volumes"
Sep 17 17:56:10.395146 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.395105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:10.428216 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428178    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"]
Sep 17 17:56:10.428574 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428559    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kube-rbac-proxy"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428576    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kube-rbac-proxy"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="storage-initializer"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="storage-initializer"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428604    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428613    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kube-rbac-proxy"
Sep 17 17:56:10.428624 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428618    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kube-rbac-proxy"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428640    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428646    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428652    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="storage-initializer"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="storage-initializer"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428725    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kube-rbac-proxy"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428736    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kserve-container"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428744    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2939886-c19b-4c70-ba4b-4afb2bbcff9c" containerName="kserve-container"
Sep 17 17:56:10.428950 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.428760    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fc7876-7bf4-4b7d-be4b-794d0df42d7d" containerName="kube-rbac-proxy"
Sep 17 17:56:10.432232 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.432215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.434646 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.434601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 17:56:10.434786 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.434696    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:56:10.434786 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.434713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:56:10.435085 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.435013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:56:10.435085 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.435046    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:56:10.435266 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.435118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 17:56:10.435266 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.435118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:56:10.444655 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.444630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"]
Sep 17 17:56:10.520405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.520365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.520405 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.520411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.520626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.520433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be456a72-73be-4901-a79b-2e349fa19934-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.520626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.520478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.520626 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.520502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnzf\" (UniqueName: \"kubernetes.io/projected/be456a72-73be-4901-a79b-2e349fa19934-kube-api-access-cpnzf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.621709 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.621673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.621914 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.621733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.621914 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:56:10.621841    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 17:56:10.622034 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.621843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be456a72-73be-4901-a79b-2e349fa19934-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.622034 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:56:10.621913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls podName:be456a72-73be-4901-a79b-2e349fa19934 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:11.121889878 +0000 UTC m=+1256.649671717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" (UID: "be456a72-73be-4901-a79b-2e349fa19934") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 17:56:10.622034 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.621974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.622213 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.622039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnzf\" (UniqueName: \"kubernetes.io/projected/be456a72-73be-4901-a79b-2e349fa19934-kube-api-access-cpnzf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.622453 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.622430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be456a72-73be-4901-a79b-2e349fa19934-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.622544 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.622508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.622617 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.622602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.630597 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.630571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnzf\" (UniqueName: \"kubernetes.io/projected/be456a72-73be-4901-a79b-2e349fa19934-kube-api-access-cpnzf\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:10.930903 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:10.930871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:11.125678 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:11.125637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:11.128270 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:11.128231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-6xlh4\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:11.343847 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:11.343725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:11.620114 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:11.620086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"]
Sep 17 17:56:11.622737 ip-10-0-135-202 kubenswrapper[2563]: W0917 17:56:11.622708    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe456a72_73be_4901_a79b_2e349fa19934.slice/crio-6ee76190f2fd20f79fae74930dfef31efd7e92520b5d3988df5d02f602a54e1d WatchSource:0}: Error finding container 6ee76190f2fd20f79fae74930dfef31efd7e92520b5d3988df5d02f602a54e1d: Status 404 returned error can't find the container with id 6ee76190f2fd20f79fae74930dfef31efd7e92520b5d3988df5d02f602a54e1d
Sep 17 17:56:11.874918 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:11.874827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerStarted","Data":"9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2"}
Sep 17 17:56:11.874918 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:11.874912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerStarted","Data":"6ee76190f2fd20f79fae74930dfef31efd7e92520b5d3988df5d02f602a54e1d"}
Sep 17 17:56:42.022110 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:42.022075    2563 generic.go:356] "Generic (PLEG): container finished" podID="be456a72-73be-4901-a79b-2e349fa19934" containerID="9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2" exitCode=0
Sep 17 17:56:42.022529 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:42.022147    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerDied","Data":"9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2"}
Sep 17 17:56:43.030112 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:43.030081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerStarted","Data":"a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0"}
Sep 17 17:56:43.030473 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:43.030140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerStarted","Data":"d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a"}
Sep 17 17:56:43.030473 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:43.030334    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:43.056522 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:43.056475    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" podStartSLOduration=3.7064530319999998 podStartE2EDuration="33.056460523s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.350007491s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:43.054190423 +0000 UTC m=+1288.581972253" watchObservedRunningTime="2026-09-17 17:56:43.056460523 +0000 UTC m=+1288.584242349"
Sep 17 17:56:44.035546 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:44.035507    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:56:44.475054 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:44.475023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:49.632936 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:49.632893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:49.769159 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:49.769122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:56:50.045931 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:56:50.045899    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:58:18.990063 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:18.990028    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:58:19.983506 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:19.983467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:24.914750 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:24.914716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:25.014440 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:25.014406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:26.703463 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:26.703426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"]
Sep 17 17:58:26.703890 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:26.703706    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kserve-container" containerID="cri-o://d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a" gracePeriod=30
Sep 17 17:58:26.703890 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:26.703719    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kube-rbac-proxy" containerID="cri-o://a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0" gracePeriod=30
Sep 17 17:58:26.726868 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:26.726840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:27.054521 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:27.054430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:27.531154 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:27.531121    2563 generic.go:356] "Generic (PLEG): container finished" podID="be456a72-73be-4901-a79b-2e349fa19934" containerID="a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0" exitCode=2
Sep 17 17:58:27.531351 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:27.531198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerDied","Data":"a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0"}
Sep 17 17:58:28.267503 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.267481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:58:28.317956 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.317923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-cabundle-cert\") pod \"be456a72-73be-4901-a79b-2e349fa19934\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") "
Sep 17 17:58:28.318156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.317975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpnzf\" (UniqueName: \"kubernetes.io/projected/be456a72-73be-4901-a79b-2e349fa19934-kube-api-access-cpnzf\") pod \"be456a72-73be-4901-a79b-2e349fa19934\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") "
Sep 17 17:58:28.318156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.318005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-kube-rbac-proxy-sar-config\") pod \"be456a72-73be-4901-a79b-2e349fa19934\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") "
Sep 17 17:58:28.318156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.318066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls\") pod \"be456a72-73be-4901-a79b-2e349fa19934\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") "
Sep 17 17:58:28.318156 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.318115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be456a72-73be-4901-a79b-2e349fa19934-kserve-provision-location\") pod \"be456a72-73be-4901-a79b-2e349fa19934\" (UID: \"be456a72-73be-4901-a79b-2e349fa19934\") "
Sep 17 17:58:28.318511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.318379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be456a72-73be-4901-a79b-2e349fa19934" (UID: "be456a72-73be-4901-a79b-2e349fa19934"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:28.318511 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.318434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be456a72-73be-4901-a79b-2e349fa19934" (UID: "be456a72-73be-4901-a79b-2e349fa19934"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:28.320224 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.320191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be456a72-73be-4901-a79b-2e349fa19934" (UID: "be456a72-73be-4901-a79b-2e349fa19934"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:28.320389 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.320375    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be456a72-73be-4901-a79b-2e349fa19934-kube-api-access-cpnzf" (OuterVolumeSpecName: "kube-api-access-cpnzf") pod "be456a72-73be-4901-a79b-2e349fa19934" (UID: "be456a72-73be-4901-a79b-2e349fa19934"). InnerVolumeSpecName "kube-api-access-cpnzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:28.325312 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.325264    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be456a72-73be-4901-a79b-2e349fa19934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be456a72-73be-4901-a79b-2e349fa19934" (UID: "be456a72-73be-4901-a79b-2e349fa19934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:28.419081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.418965    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be456a72-73be-4901-a79b-2e349fa19934-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:58:28.419081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.419016    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be456a72-73be-4901-a79b-2e349fa19934-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:58:28.419081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.419032    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:58:28.419081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.419044    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnzf\" (UniqueName: \"kubernetes.io/projected/be456a72-73be-4901-a79b-2e349fa19934-kube-api-access-cpnzf\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:58:28.419081 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.419056    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be456a72-73be-4901-a79b-2e349fa19934-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 17:58:28.537681 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.537644    2563 generic.go:356] "Generic (PLEG): container finished" podID="be456a72-73be-4901-a79b-2e349fa19934" containerID="d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a" exitCode=0
Sep 17 17:58:28.537867 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.537705    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"
Sep 17 17:58:28.537867 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.537729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerDied","Data":"d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a"}
Sep 17 17:58:28.537867 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.537810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4" event={"ID":"be456a72-73be-4901-a79b-2e349fa19934","Type":"ContainerDied","Data":"6ee76190f2fd20f79fae74930dfef31efd7e92520b5d3988df5d02f602a54e1d"}
Sep 17 17:58:28.537867 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.537840    2563 scope.go:117] "RemoveContainer" containerID="a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0"
Sep 17 17:58:28.547421 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.547401    2563 scope.go:117] "RemoveContainer" containerID="d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a"
Sep 17 17:58:28.555653 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.555631    2563 scope.go:117] "RemoveContainer" containerID="9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2"
Sep 17 17:58:28.561472 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.561447    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"]
Sep 17 17:58:28.564520 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.564495    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-6xlh4"]
Sep 17 17:58:28.565074 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.565057    2563 scope.go:117] "RemoveContainer" containerID="a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0"
Sep 17 17:58:28.565404 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:58:28.565384    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0\": container with ID starting with a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0 not found: ID does not exist" containerID="a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0"
Sep 17 17:58:28.565475 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.565419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0"} err="failed to get container status \"a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0\": rpc error: code = NotFound desc = could not find container \"a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0\": container with ID starting with a9437499b77ab70876b744285eac38b15c53d74e6653780edfb4e6481a0947d0 not found: ID does not exist"
Sep 17 17:58:28.565475 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.565446    2563 scope.go:117] "RemoveContainer" containerID="d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a"
Sep 17 17:58:28.565723 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:58:28.565705    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a\": container with ID starting with d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a not found: ID does not exist" containerID="d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a"
Sep 17 17:58:28.565768 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.565731    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a"} err="failed to get container status \"d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a\": rpc error: code = NotFound desc = could not find container \"d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a\": container with ID starting with d343e07125ae41e70f62efc40b855c936ee8320482ea30ec93b1c9388c11360a not found: ID does not exist"
Sep 17 17:58:28.565768 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.565748    2563 scope.go:117] "RemoveContainer" containerID="9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2"
Sep 17 17:58:28.566016 ip-10-0-135-202 kubenswrapper[2563]: E0917 17:58:28.565996    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2\": container with ID starting with 9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2 not found: ID does not exist" containerID="9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2"
Sep 17 17:58:28.566065 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.566022    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2"} err="failed to get container status \"9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2\": rpc error: code = NotFound desc = could not find container \"9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2\": container with ID starting with 9aab3d296dd1437f85f71cef9598504bfd8afdb2d078249ed68eb185225e7cb2 not found: ID does not exist"
Sep 17 17:58:28.958848 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:28.958808    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be456a72-73be-4901-a79b-2e349fa19934" path="/var/lib/kubelet/pods/be456a72-73be-4901-a79b-2e349fa19934/volumes"
Sep 17 17:58:31.857399 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:31.857363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:32.020083 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:32.020047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.219752 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:59.219716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:59.379091 ip-10-0-135-202 kubenswrapper[2563]: I0917 17:58:59.379052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:33.232159 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:33.232022    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:00:33.236260 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:33.233003    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:00:39.421915 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:39.421826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:44.604562 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.604526    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:44.685547 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.685510    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"]
Sep 17 18:00:44.685972 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.685956    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kserve-container"
Sep 17 18:00:44.685972 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.685974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kserve-container"
Sep 17 18:00:44.686080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.685988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="storage-initializer"
Sep 17 18:00:44.686080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.685997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="storage-initializer"
Sep 17 18:00:44.686080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.686016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kube-rbac-proxy"
Sep 17 18:00:44.686080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.686023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kube-rbac-proxy"
Sep 17 18:00:44.686214 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.686106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kube-rbac-proxy"
Sep 17 18:00:44.686214 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.686115    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="be456a72-73be-4901-a79b-2e349fa19934" containerName="kserve-container"
Sep 17 18:00:44.689512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.689494    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.691863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.691831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:00:44.691863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.691854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:00:44.692010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.691858    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:00:44.692010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.691872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 18:00:44.692512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.692488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:00:44.692727 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.692706    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:00:44.692872 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.692709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:00:44.701128 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.701109    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"]
Sep 17 18:00:44.790897 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.790864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:00:44.802973 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.802951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.803096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.802991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzcg\" (UniqueName: \"kubernetes.io/projected/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-api-access-cxzcg\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.803096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.803030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.803096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.803055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.803096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.803080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.903853 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.903743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.903853 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.903816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzcg\" (UniqueName: \"kubernetes.io/projected/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-api-access-cxzcg\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.904066 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.903871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.904066 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.903910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.904066 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.903985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.904393 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.904362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.904641 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.904619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.904712 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.904638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.906322 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.906296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:44.913454 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:44.913430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzcg\" (UniqueName: \"kubernetes.io/projected/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-api-access-cxzcg\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:45.000449 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:45.000413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:00:45.275391 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:45.275360    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"]
Sep 17 18:00:45.279358 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:00:45.279323    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab506e4c_9b3d_4966_9805_7ed29dd00b8a.slice/crio-1c7c1d6a7ccc44672a1681f8931f1bf092ff91b0764e79a4bfe6daf20c29003d WatchSource:0}: Error finding container 1c7c1d6a7ccc44672a1681f8931f1bf092ff91b0764e79a4bfe6daf20c29003d: Status 404 returned error can't find the container with id 1c7c1d6a7ccc44672a1681f8931f1bf092ff91b0764e79a4bfe6daf20c29003d
Sep 17 18:00:46.163951 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:46.163912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerStarted","Data":"1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705"}
Sep 17 18:00:46.164425 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:46.163981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerStarted","Data":"1c7c1d6a7ccc44672a1681f8931f1bf092ff91b0764e79a4bfe6daf20c29003d"}
Sep 17 18:00:50.183876 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:50.183840    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerID="1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705" exitCode=0
Sep 17 18:00:50.184353 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:50.183918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerDied","Data":"1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705"}
Sep 17 18:00:50.185169 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:00:50.185152    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:01:01.713228 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:01:01.713173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:41.448600 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:41.448507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:43.602069 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:43.602010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:43.820052 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:43.820014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:51.873180 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:51.873143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerStarted","Data":"48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1"}
Sep 17 18:02:51.873638 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:51.873216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:02:51.873638 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:51.873243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:02:51.873638 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:51.873262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerStarted","Data":"07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2"}
Sep 17 18:02:51.894547 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:51.894476    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" podStartSLOduration=2.87664306 podStartE2EDuration="2m7.894403768s" totalImagesPullingTime="2m1.088001925s" totalInitContainerRuntime="3.929758783s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:51.893902205 +0000 UTC m=+1657.421684028" watchObservedRunningTime="2026-09-17 18:02:51.894403768 +0000 UTC m=+1657.422185594"
Sep 17 18:02:57.883205 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:02:57.883175    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:03:27.936202 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:27.936161    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:03:37.886039 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:37.885964    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:03:44.693167 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:44.693121    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"]
Sep 17 18:03:44.693727 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:44.693443    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kserve-container" containerID="cri-o://07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2" gracePeriod=30
Sep 17 18:03:44.693727 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:44.693503    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kube-rbac-proxy" containerID="cri-o://48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1" gracePeriod=30
Sep 17 18:03:44.715110 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:44.715071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:45.121097 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:45.121002    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerID="48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1" exitCode=2
Sep 17 18:03:45.121097 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:45.121076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerDied","Data":"48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1"}
Sep 17 18:03:47.878712 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:47.878667    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 18:03:49.883026 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:49.882975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:50.010293 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:50.010262    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:52.536280 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.536244    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:03:52.579722 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.579690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kserve-provision-location\") pod \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") "
Sep 17 18:03:52.579934 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.579738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxzcg\" (UniqueName: \"kubernetes.io/projected/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-api-access-cxzcg\") pod \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") "
Sep 17 18:03:52.579934 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.579771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-rbac-proxy-sar-config\") pod \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") "
Sep 17 18:03:52.579934 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.579828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-cabundle-cert\") pod \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") "
Sep 17 18:03:52.579934 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.579873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-proxy-tls\") pod \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\" (UID: \"ab506e4c-9b3d-4966-9805-7ed29dd00b8a\") "
Sep 17 18:03:52.580165 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.580083    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab506e4c-9b3d-4966-9805-7ed29dd00b8a" (UID: "ab506e4c-9b3d-4966-9805-7ed29dd00b8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:03:52.580227 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.580194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab506e4c-9b3d-4966-9805-7ed29dd00b8a" (UID: "ab506e4c-9b3d-4966-9805-7ed29dd00b8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:52.580286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.580245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab506e4c-9b3d-4966-9805-7ed29dd00b8a" (UID: "ab506e4c-9b3d-4966-9805-7ed29dd00b8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:03:52.581952 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.581923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-api-access-cxzcg" (OuterVolumeSpecName: "kube-api-access-cxzcg") pod "ab506e4c-9b3d-4966-9805-7ed29dd00b8a" (UID: "ab506e4c-9b3d-4966-9805-7ed29dd00b8a"). InnerVolumeSpecName "kube-api-access-cxzcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:52.582047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.581962    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab506e4c-9b3d-4966-9805-7ed29dd00b8a" (UID: "ab506e4c-9b3d-4966-9805-7ed29dd00b8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:52.680590 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.680542    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:03:52.680590 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.680592    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:03:52.680590 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.680602    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzcg\" (UniqueName: \"kubernetes.io/projected/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-api-access-cxzcg\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:03:52.680867 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.680611    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:03:52.680867 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:52.680621    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab506e4c-9b3d-4966-9805-7ed29dd00b8a-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:03:53.162097 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.162058    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerID="07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2" exitCode=0
Sep 17 18:03:53.162304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.162122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"
Sep 17 18:03:53.162304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.162133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerDied","Data":"07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2"}
Sep 17 18:03:53.162304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.162192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7" event={"ID":"ab506e4c-9b3d-4966-9805-7ed29dd00b8a","Type":"ContainerDied","Data":"1c7c1d6a7ccc44672a1681f8931f1bf092ff91b0764e79a4bfe6daf20c29003d"}
Sep 17 18:03:53.162304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.162219    2563 scope.go:117] "RemoveContainer" containerID="48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1"
Sep 17 18:03:53.170775 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.170754    2563 scope.go:117] "RemoveContainer" containerID="07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2"
Sep 17 18:03:53.178692 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.178637    2563 scope.go:117] "RemoveContainer" containerID="1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705"
Sep 17 18:03:53.181516 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.181491    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"]
Sep 17 18:03:53.183262 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.183241    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-gdjj7"]
Sep 17 18:03:53.187712 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.187693    2563 scope.go:117] "RemoveContainer" containerID="48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1"
Sep 17 18:03:53.188015 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:03:53.187994    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1\": container with ID starting with 48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1 not found: ID does not exist" containerID="48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1"
Sep 17 18:03:53.188099 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.188028    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1"} err="failed to get container status \"48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1\": rpc error: code = NotFound desc = could not find container \"48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1\": container with ID starting with 48614b2920eeabcb828f767d47ac3db52301025f138623959572f2b9672697f1 not found: ID does not exist"
Sep 17 18:03:53.188099 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.188056    2563 scope.go:117] "RemoveContainer" containerID="07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2"
Sep 17 18:03:53.188340 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:03:53.188322    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2\": container with ID starting with 07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2 not found: ID does not exist" containerID="07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2"
Sep 17 18:03:53.188379 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.188347    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2"} err="failed to get container status \"07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2\": rpc error: code = NotFound desc = could not find container \"07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2\": container with ID starting with 07c361e78ebc05a0efb49d6b40c43378acdb7883b2ab06c5b1a2e9800109fec2 not found: ID does not exist"
Sep 17 18:03:53.188379 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.188363    2563 scope.go:117] "RemoveContainer" containerID="1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705"
Sep 17 18:03:53.188603 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:03:53.188586    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705\": container with ID starting with 1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705 not found: ID does not exist" containerID="1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705"
Sep 17 18:03:53.188647 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:53.188608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705"} err="failed to get container status \"1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705\": rpc error: code = NotFound desc = could not find container \"1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705\": container with ID starting with 1bcf12299d02028de3deb5ad748ac40415264cbbfe515e7f49dc96241877a705 not found: ID does not exist"
Sep 17 18:03:54.965454 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:03:54.965416    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" path="/var/lib/kubelet/pods/ab506e4c-9b3d-4966-9805-7ed29dd00b8a/volumes"
Sep 17 18:04:21.784140 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:21.784102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:26.884769 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.884736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:26.919503 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"]
Sep 17 18:04:26.919900 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="storage-initializer"
Sep 17 18:04:26.919900 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="storage-initializer"
Sep 17 18:04:26.920089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919918    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kserve-container"
Sep 17 18:04:26.920089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919926    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kserve-container"
Sep 17 18:04:26.920089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919935    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kube-rbac-proxy"
Sep 17 18:04:26.920089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.919943    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kube-rbac-proxy"
Sep 17 18:04:26.920089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.920070    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kube-rbac-proxy"
Sep 17 18:04:26.920089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.920086    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab506e4c-9b3d-4966-9805-7ed29dd00b8a" containerName="kserve-container"
Sep 17 18:04:26.922944 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.922924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:26.925883 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.925848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:04:26.925883 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.925864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:04:26.926052 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.925920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:04:26.926052 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.925868    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:04:26.926414 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.926397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 18:04:26.926512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.926399    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:04:26.926512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.926484    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:04:26.934451 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:26.934429    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"]
Sep 17 18:04:27.089361 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.089331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.089536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.089380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.089536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.089424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-api-access-84xr7\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.089536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.089451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.089536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.089468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.115213 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.115181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:27.190400 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.190373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.190555 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.190419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.190555 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.190453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-api-access-84xr7\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.190555 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.190476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.190555 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.190493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.190808 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.190768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.191136 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.191117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.191286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.191267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.193041 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.193021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.198527 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.198509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-api-access-84xr7\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.235003 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.234979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:27.507978 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:27.507949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"]
Sep 17 18:04:27.510084 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:04:27.510053    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a4dff8_fa6f_4a0c_9741_0e2edd90f374.slice/crio-7ca251b4a9cc6aa0a38c8a946e96697367384348d27324f502dd1c63472c07b1 WatchSource:0}: Error finding container 7ca251b4a9cc6aa0a38c8a946e96697367384348d27324f502dd1c63472c07b1: Status 404 returned error can't find the container with id 7ca251b4a9cc6aa0a38c8a946e96697367384348d27324f502dd1c63472c07b1
Sep 17 18:04:28.328250 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:28.328206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerStarted","Data":"6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841"}
Sep 17 18:04:28.328647 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:28.328284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerStarted","Data":"7ca251b4a9cc6aa0a38c8a946e96697367384348d27324f502dd1c63472c07b1"}
Sep 17 18:04:32.349095 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:32.349059    2563 generic.go:356] "Generic (PLEG): container finished" podID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerID="6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841" exitCode=0
Sep 17 18:04:32.349486 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:32.349134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerDied","Data":"6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841"}
Sep 17 18:04:33.356255 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:33.356221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerStarted","Data":"73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c"}
Sep 17 18:04:33.356754 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:33.356289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerStarted","Data":"d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd"}
Sep 17 18:04:33.356754 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:33.356486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:33.356754 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:33.356528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:04:33.382987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:33.382938    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podStartSLOduration=3.441469868 podStartE2EDuration="7.382922583s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941452715s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:33.380561525 +0000 UTC m=+1758.908343367" watchObservedRunningTime="2026-09-17 18:04:33.382922583 +0000 UTC m=+1758.910704409"
Sep 17 18:04:39.042366 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:39.042329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:39.366655 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:04:39.366578    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:05:09.436125 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:09.436033    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:05:09.844291 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:09.844203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.969837 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:14.969787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.993847 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:14.993817    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"]
Sep 17 18:05:14.996527 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:14.996509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:14.999050 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:14.999018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 18:05:14.999207 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:14.999054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:05:15.009923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.009898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"]
Sep 17 18:05:15.103593 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.103562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.103826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.103622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.103826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.103669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0640893c-9885-439b-a2e2-e168b5998405-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.103826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.103711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czs6k\" (UniqueName: \"kubernetes.io/projected/0640893c-9885-439b-a2e2-e168b5998405-kube-api-access-czs6k\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.103826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.103737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0640893c-9885-439b-a2e2-e168b5998405-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.204764 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.204737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.204974 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.204779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0640893c-9885-439b-a2e2-e168b5998405-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.204974 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.204835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czs6k\" (UniqueName: \"kubernetes.io/projected/0640893c-9885-439b-a2e2-e168b5998405-kube-api-access-czs6k\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.204974 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.204853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0640893c-9885-439b-a2e2-e168b5998405-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.204974 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.204902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.205271 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.205246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0640893c-9885-439b-a2e2-e168b5998405-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.205513 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.205493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.205564 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.205524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.207469 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.207443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0640893c-9885-439b-a2e2-e168b5998405-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.215054 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.215032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czs6k\" (UniqueName: \"kubernetes.io/projected/0640893c-9885-439b-a2e2-e168b5998405-kube-api-access-czs6k\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.267233 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.267144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:15.308559 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.308527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:15.588472 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:15.588446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"]
Sep 17 18:05:15.590529 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:05:15.590491    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0640893c_9885_439b_a2e2_e168b5998405.slice/crio-6bdbe97b376e48770279f97a88fab33232004489cb53c14a024d90ea7cb001ec WatchSource:0}: Error finding container 6bdbe97b376e48770279f97a88fab33232004489cb53c14a024d90ea7cb001ec: Status 404 returned error can't find the container with id 6bdbe97b376e48770279f97a88fab33232004489cb53c14a024d90ea7cb001ec
Sep 17 18:05:16.559546 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:16.559499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerStarted","Data":"980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc"}
Sep 17 18:05:16.559960 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:16.559596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerStarted","Data":"6bdbe97b376e48770279f97a88fab33232004489cb53c14a024d90ea7cb001ec"}
Sep 17 18:05:19.369807 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:19.369767    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:05:25.601728 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:25.601693    2563 generic.go:356] "Generic (PLEG): container finished" podID="0640893c-9885-439b-a2e2-e168b5998405" containerID="980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc" exitCode=0
Sep 17 18:05:25.602139 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:25.601778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerDied","Data":"980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc"}
Sep 17 18:05:26.609236 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.609197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerStarted","Data":"413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2"}
Sep 17 18:05:26.609617 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.609267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerStarted","Data":"9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed"}
Sep 17 18:05:26.609617 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.609488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:26.634758 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.634709    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podStartSLOduration=3.578471576 podStartE2EDuration="12.634686509s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.056214933s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:26.632699823 +0000 UTC m=+1812.160481661" watchObservedRunningTime="2026-09-17 18:05:26.634686509 +0000 UTC m=+1812.162468335"
Sep 17 18:05:26.996969 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.996934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"]
Sep 17 18:05:26.997469 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.997438    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kserve-container" containerID="cri-o://d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd" gracePeriod=30
Sep 17 18:05:26.997684 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:26.997486    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kube-rbac-proxy" containerID="cri-o://73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c" gracePeriod=30
Sep 17 18:05:27.025010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:27.024973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:27.617026 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:27.616984    2563 generic.go:356] "Generic (PLEG): container finished" podID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerID="73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c" exitCode=2
Sep 17 18:05:27.617386 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:27.617053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerDied","Data":"73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c"}
Sep 17 18:05:27.617424 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:27.617414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:27.618813 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:27.618779    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:05:28.622498 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:28.622439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:05:29.362186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:29.362137    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 18:05:30.408058 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:30.408006    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.42:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:05:32.215763 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:32.215728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:32.327049 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:32.327013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:33.266426 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:33.266308    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:05:33.282307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:33.269604    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:05:33.627671 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:33.627588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:33.628243 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:33.628209    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:05:34.361672 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:34.361622    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 18:05:35.057647 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.057619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:05:35.098075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.097992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-rbac-proxy-sar-config\") pod \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") "
Sep 17 18:05:35.098075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-cabundle-cert\") pod \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") "
Sep 17 18:05:35.098301 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-proxy-tls\") pod \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") "
Sep 17 18:05:35.098301 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kserve-provision-location\") pod \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") "
Sep 17 18:05:35.098301 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-api-access-84xr7\") pod \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\" (UID: \"05a4dff8-fa6f-4a0c-9741-0e2edd90f374\") "
Sep 17 18:05:35.098448 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05a4dff8-fa6f-4a0c-9741-0e2edd90f374" (UID: "05a4dff8-fa6f-4a0c-9741-0e2edd90f374"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:35.098448 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05a4dff8-fa6f-4a0c-9741-0e2edd90f374" (UID: "05a4dff8-fa6f-4a0c-9741-0e2edd90f374"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:35.098613 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05a4dff8-fa6f-4a0c-9741-0e2edd90f374" (UID: "05a4dff8-fa6f-4a0c-9741-0e2edd90f374"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:35.098682 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098614    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:35.098682 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.098633    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:35.100386 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.100360    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-api-access-84xr7" (OuterVolumeSpecName: "kube-api-access-84xr7") pod "05a4dff8-fa6f-4a0c-9741-0e2edd90f374" (UID: "05a4dff8-fa6f-4a0c-9741-0e2edd90f374"). InnerVolumeSpecName "kube-api-access-84xr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:35.100714 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.100698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05a4dff8-fa6f-4a0c-9741-0e2edd90f374" (UID: "05a4dff8-fa6f-4a0c-9741-0e2edd90f374"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:35.199386 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.199346    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:35.199386 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.199378    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:35.199386 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.199393    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84xr7\" (UniqueName: \"kubernetes.io/projected/05a4dff8-fa6f-4a0c-9741-0e2edd90f374-kube-api-access-84xr7\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:35.659633 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.659596    2563 generic.go:356] "Generic (PLEG): container finished" podID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerID="d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd" exitCode=0
Sep 17 18:05:35.660120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.659649    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"
Sep 17 18:05:35.660120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.659680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerDied","Data":"d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd"}
Sep 17 18:05:35.660120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.659758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6" event={"ID":"05a4dff8-fa6f-4a0c-9741-0e2edd90f374","Type":"ContainerDied","Data":"7ca251b4a9cc6aa0a38c8a946e96697367384348d27324f502dd1c63472c07b1"}
Sep 17 18:05:35.660120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.659818    2563 scope.go:117] "RemoveContainer" containerID="73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c"
Sep 17 18:05:35.669970 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.669948    2563 scope.go:117] "RemoveContainer" containerID="d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd"
Sep 17 18:05:35.680257 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.680234    2563 scope.go:117] "RemoveContainer" containerID="6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841"
Sep 17 18:05:35.683596 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.683573    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"]
Sep 17 18:05:35.688101 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.688078    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-75gr6"]
Sep 17 18:05:35.690777 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.690751    2563 scope.go:117] "RemoveContainer" containerID="73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c"
Sep 17 18:05:35.691107 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:05:35.691088    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c\": container with ID starting with 73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c not found: ID does not exist" containerID="73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c"
Sep 17 18:05:35.691182 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.691117    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c"} err="failed to get container status \"73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c\": rpc error: code = NotFound desc = could not find container \"73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c\": container with ID starting with 73f1fa13dd5a445404c75d146bbf47d44e39b2cdc52a9d51ed12a378e3de7b9c not found: ID does not exist"
Sep 17 18:05:35.691182 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.691150    2563 scope.go:117] "RemoveContainer" containerID="d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd"
Sep 17 18:05:35.691445 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:05:35.691430    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd\": container with ID starting with d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd not found: ID does not exist" containerID="d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd"
Sep 17 18:05:35.691500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.691447    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd"} err="failed to get container status \"d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd\": rpc error: code = NotFound desc = could not find container \"d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd\": container with ID starting with d762fc6554d3494c1c3ee45df06af8d4ecef3efb9ac87adc3006e46e765f56fd not found: ID does not exist"
Sep 17 18:05:35.691500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.691459    2563 scope.go:117] "RemoveContainer" containerID="6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841"
Sep 17 18:05:35.691730 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:05:35.691714    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841\": container with ID starting with 6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841 not found: ID does not exist" containerID="6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841"
Sep 17 18:05:35.691768 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:35.691736    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841"} err="failed to get container status \"6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841\": rpc error: code = NotFound desc = could not find container \"6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841\": container with ID starting with 6611b34e1d34a2143a02bf53cb5ea3774b0eaf5f76c4ae0ff80e06149e8a3841 not found: ID does not exist"
Sep 17 18:05:36.959648 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:36.959608    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" path="/var/lib/kubelet/pods/05a4dff8-fa6f-4a0c-9741-0e2edd90f374/volumes"
Sep 17 18:05:43.628965 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:43.628936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:57.818077 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:57.818044    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"]
Sep 17 18:05:57.818550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:57.818392    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kube-rbac-proxy" containerID="cri-o://413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2" gracePeriod=30
Sep 17 18:05:57.818550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:57.818368    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container" containerID="cri-o://9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed" gracePeriod=30
Sep 17 18:05:57.845176 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:57.845144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:57.945029 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:57.945000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:58.623399 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:58.623344    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:05:58.777968 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:58.777934    2563 generic.go:356] "Generic (PLEG): container finished" podID="0640893c-9885-439b-a2e2-e168b5998405" containerID="413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2" exitCode=2
Sep 17 18:05:58.778151 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:58.778007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerDied","Data":"413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2"}
Sep 17 18:05:59.559907 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.559884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:59.623471 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0640893c-9885-439b-a2e2-e168b5998405-proxy-tls\") pod \"0640893c-9885-439b-a2e2-e168b5998405\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") "
Sep 17 18:05:59.623626 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-kube-rbac-proxy-sar-config\") pod \"0640893c-9885-439b-a2e2-e168b5998405\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") "
Sep 17 18:05:59.623626 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-cabundle-cert\") pod \"0640893c-9885-439b-a2e2-e168b5998405\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") "
Sep 17 18:05:59.623626 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623610    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czs6k\" (UniqueName: \"kubernetes.io/projected/0640893c-9885-439b-a2e2-e168b5998405-kube-api-access-czs6k\") pod \"0640893c-9885-439b-a2e2-e168b5998405\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") "
Sep 17 18:05:59.623834 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0640893c-9885-439b-a2e2-e168b5998405-kserve-provision-location\") pod \"0640893c-9885-439b-a2e2-e168b5998405\" (UID: \"0640893c-9885-439b-a2e2-e168b5998405\") "
Sep 17 18:05:59.624010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0640893c-9885-439b-a2e2-e168b5998405" (UID: "0640893c-9885-439b-a2e2-e168b5998405"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:59.624010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.623998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0640893c-9885-439b-a2e2-e168b5998405" (UID: "0640893c-9885-439b-a2e2-e168b5998405"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:59.624764 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.624582    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0640893c-9885-439b-a2e2-e168b5998405-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0640893c-9885-439b-a2e2-e168b5998405" (UID: "0640893c-9885-439b-a2e2-e168b5998405"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:59.631961 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.631935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0640893c-9885-439b-a2e2-e168b5998405-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0640893c-9885-439b-a2e2-e168b5998405" (UID: "0640893c-9885-439b-a2e2-e168b5998405"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:59.632075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.631972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0640893c-9885-439b-a2e2-e168b5998405-kube-api-access-czs6k" (OuterVolumeSpecName: "kube-api-access-czs6k") pod "0640893c-9885-439b-a2e2-e168b5998405" (UID: "0640893c-9885-439b-a2e2-e168b5998405"). InnerVolumeSpecName "kube-api-access-czs6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:59.724899 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.724870    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.724899 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.724896    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0640893c-9885-439b-a2e2-e168b5998405-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.725083 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.724907    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czs6k\" (UniqueName: \"kubernetes.io/projected/0640893c-9885-439b-a2e2-e168b5998405-kube-api-access-czs6k\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.725083 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.724916    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0640893c-9885-439b-a2e2-e168b5998405-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.725083 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.724926    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0640893c-9885-439b-a2e2-e168b5998405-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:05:59.784598 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.784563    2563 generic.go:356] "Generic (PLEG): container finished" podID="0640893c-9885-439b-a2e2-e168b5998405" containerID="9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed" exitCode=0
Sep 17 18:05:59.784750 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.784639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"
Sep 17 18:05:59.784750 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.784644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerDied","Data":"9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed"}
Sep 17 18:05:59.784750 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.784744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk" event={"ID":"0640893c-9885-439b-a2e2-e168b5998405","Type":"ContainerDied","Data":"6bdbe97b376e48770279f97a88fab33232004489cb53c14a024d90ea7cb001ec"}
Sep 17 18:05:59.784881 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.784778    2563 scope.go:117] "RemoveContainer" containerID="413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2"
Sep 17 18:05:59.794229 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.794212    2563 scope.go:117] "RemoveContainer" containerID="9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed"
Sep 17 18:05:59.801922 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.801906    2563 scope.go:117] "RemoveContainer" containerID="980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc"
Sep 17 18:05:59.807970 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.807945    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"]
Sep 17 18:05:59.809883 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.809868    2563 scope.go:117] "RemoveContainer" containerID="413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2"
Sep 17 18:05:59.810178 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:05:59.810153    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2\": container with ID starting with 413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2 not found: ID does not exist" containerID="413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2"
Sep 17 18:05:59.810228 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.810188    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2"} err="failed to get container status \"413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2\": rpc error: code = NotFound desc = could not find container \"413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2\": container with ID starting with 413d8bf4342d85c5bd000704f8ef002458573d0cccef647509391e17ef32a7b2 not found: ID does not exist"
Sep 17 18:05:59.810228 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.810205    2563 scope.go:117] "RemoveContainer" containerID="9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed"
Sep 17 18:05:59.810439 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:05:59.810423    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed\": container with ID starting with 9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed not found: ID does not exist" containerID="9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed"
Sep 17 18:05:59.810480 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.810441    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed"} err="failed to get container status \"9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed\": rpc error: code = NotFound desc = could not find container \"9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed\": container with ID starting with 9721406e6e66198e99e9fe837f164714fd742996efbf1720875f93b26d70a0ed not found: ID does not exist"
Sep 17 18:05:59.810480 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.810454    2563 scope.go:117] "RemoveContainer" containerID="980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc"
Sep 17 18:05:59.810670 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:05:59.810653    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc\": container with ID starting with 980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc not found: ID does not exist" containerID="980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc"
Sep 17 18:05:59.810729 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.810687    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc"} err="failed to get container status \"980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc\": rpc error: code = NotFound desc = could not find container \"980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc\": container with ID starting with 980398c9ce2fc50188976ee823d4dc5bb41e709b9d5917a8c6349c74227578cc not found: ID does not exist"
Sep 17 18:05:59.814188 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:05:59.814165    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gv2gk"]
Sep 17 18:06:00.957965 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:00.957930    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0640893c-9885-439b-a2e2-e168b5998405" path="/var/lib/kubelet/pods/0640893c-9885-439b-a2e2-e168b5998405/volumes"
Sep 17 18:06:02.986041 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:02.986009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:03.090442 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:03.090409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:38.340203 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:38.340115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:43.458742 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.458702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:43.552510 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.552471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:43.583661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.583625    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"]
Sep 17 18:06:43.584191 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584171    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kserve-container"
Sep 17 18:06:43.584191 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584191    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kserve-container"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584206    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="storage-initializer"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="storage-initializer"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="storage-initializer"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584237    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="storage-initializer"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584248    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kube-rbac-proxy"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584256    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kube-rbac-proxy"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kube-rbac-proxy"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584285    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kube-rbac-proxy"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584311    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container"
Sep 17 18:06:43.584328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container"
Sep 17 18:06:43.584632 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kserve-container"
Sep 17 18:06:43.584632 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584398    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0640893c-9885-439b-a2e2-e168b5998405" containerName="kube-rbac-proxy"
Sep 17 18:06:43.584632 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584411    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kube-rbac-proxy"
Sep 17 18:06:43.584632 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.584420    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a4dff8-fa6f-4a0c-9741-0e2edd90f374" containerName="kserve-container"
Sep 17 18:06:43.588118 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.588100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.590457 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.590435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:06:43.590665 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.590649    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:06:43.591007 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.590977    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:06:43.591145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.591016    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:06:43.591145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.590985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:06:43.591145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.591050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:06:43.591145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.591094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:06:43.599609 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.599588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"]
Sep 17 18:06:43.660323 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.660288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:43.716859 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.716755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.717030 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.716904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.717030 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.716942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sz75\" (UniqueName: \"kubernetes.io/projected/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-api-access-9sz75\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.717030 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.716973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.717144 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.717059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.818166 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.818122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.818166 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.818166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sz75\" (UniqueName: \"kubernetes.io/projected/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-api-access-9sz75\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.818432 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.818188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.818432 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.818225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.818432 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.818247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.818432 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:06:43.818395    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:06:43.818722 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:06:43.818468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls podName:48de669d-e0a7-4f8b-9f02-37f4d59ddd65 nodeName:}" failed. No retries permitted until 2026-09-17 18:06:44.318447919 +0000 UTC m=+1889.846229725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls") pod "isvc-canary-create-predictor-7d998654b-9ssf4" (UID: "48de669d-e0a7-4f8b-9f02-37f4d59ddd65") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:06:43.818722 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.818645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.819066 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.819043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.819066 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.819041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:43.830679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:43.830653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sz75\" (UniqueName: \"kubernetes.io/projected/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-api-access-9sz75\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:44.077649 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.077562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:44.323365 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.323328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:44.325814 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.325757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-9ssf4\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:44.501993 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.501953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:06:44.778698 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.778673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"]
Sep 17 18:06:44.780593 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:06:44.780562    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48de669d_e0a7_4f8b_9f02_37f4d59ddd65.slice/crio-984b88247000a64d38fc2042af658ece29094f0053b8d15c18431b866b719f8b WatchSource:0}: Error finding container 984b88247000a64d38fc2042af658ece29094f0053b8d15c18431b866b719f8b: Status 404 returned error can't find the container with id 984b88247000a64d38fc2042af658ece29094f0053b8d15c18431b866b719f8b
Sep 17 18:06:44.998015 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.997973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerStarted","Data":"77f049467ac9eb4858b1f6a17a891f3f02e82da969739f6990cd4e8620eb32d2"}
Sep 17 18:06:44.998212 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:44.998044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerStarted","Data":"984b88247000a64d38fc2042af658ece29094f0053b8d15c18431b866b719f8b"}
Sep 17 18:06:49.016696 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:49.016606    2563 generic.go:356] "Generic (PLEG): container finished" podID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerID="77f049467ac9eb4858b1f6a17a891f3f02e82da969739f6990cd4e8620eb32d2" exitCode=0
Sep 17 18:06:49.016696 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:49.016679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerDied","Data":"77f049467ac9eb4858b1f6a17a891f3f02e82da969739f6990cd4e8620eb32d2"}
Sep 17 18:06:49.017928 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:06:49.017914    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:07:02.102800 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:02.102758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerStarted","Data":"78a0df99c4845283831fe2218b8febdd7613db2bdc23a6ffc4187e02cd8445e9"}
Sep 17 18:07:02.103116 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:02.102834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerStarted","Data":"485b8af7af0c55c76a4c9f13f8070e4d15750848f2915e1fbbfbdadcf9dca9d0"}
Sep 17 18:07:02.103116 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:02.103025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:07:02.126231 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:02.126186    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podStartSLOduration=2.322746113 podStartE2EDuration="19.126171231s" totalImagesPullingTime="12.94859758s" totalInitContainerRuntime="3.854827538s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:02.123878955 +0000 UTC m=+1907.651660781" watchObservedRunningTime="2026-09-17 18:07:02.126171231 +0000 UTC m=+1907.653953056"
Sep 17 18:07:02.392908 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:02.392786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:03.107961 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:03.107924    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:07:03.109394 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:03.109364    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:04.112904 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:04.112862    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:07.572500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:07.572465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:07.842231 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:07.842148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:09.117952 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:09.117918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:07:09.118584 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:09.118554    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:19.119375 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:19.119330    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:29.118787 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:29.118736    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:39.119415 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:39.119374    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:49.119178 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:49.119133    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:07:59.119240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:07:59.119190    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:08:09.119048 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:09.118958    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:08:19.119985 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:19.119954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:08:23.592718 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:23.592668    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"]
Sep 17 18:08:23.593119 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:23.593014    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container" containerID="cri-o://485b8af7af0c55c76a4c9f13f8070e4d15750848f2915e1fbbfbdadcf9dca9d0" gracePeriod=30
Sep 17 18:08:23.593119 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:23.593096    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kube-rbac-proxy" containerID="cri-o://78a0df99c4845283831fe2218b8febdd7613db2bdc23a6ffc4187e02cd8445e9" gracePeriod=30
Sep 17 18:08:23.608184 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:23.608155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:23.721411 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:23.721377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:24.113461 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:24.113414    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 18:08:24.494759 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:24.494727    2563 generic.go:356] "Generic (PLEG): container finished" podID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerID="78a0df99c4845283831fe2218b8febdd7613db2bdc23a6ffc4187e02cd8445e9" exitCode=2
Sep 17 18:08:24.494944 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:24.494768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerDied","Data":"78a0df99c4845283831fe2218b8febdd7613db2bdc23a6ffc4187e02cd8445e9"}
Sep 17 18:08:28.515385 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.515356    2563 generic.go:356] "Generic (PLEG): container finished" podID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerID="485b8af7af0c55c76a4c9f13f8070e4d15750848f2915e1fbbfbdadcf9dca9d0" exitCode=0
Sep 17 18:08:28.515712 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.515402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerDied","Data":"485b8af7af0c55c76a4c9f13f8070e4d15750848f2915e1fbbfbdadcf9dca9d0"}
Sep 17 18:08:28.531982 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.531956    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:08:28.670204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.670128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sz75\" (UniqueName: \"kubernetes.io/projected/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-api-access-9sz75\") pod \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") "
Sep 17 18:08:28.670204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.670197    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls\") pod \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") "
Sep 17 18:08:28.670474 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.670217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-rbac-proxy-sar-config\") pod \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") "
Sep 17 18:08:28.670474 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.670255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-cabundle-cert\") pod \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") "
Sep 17 18:08:28.670474 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.670374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kserve-provision-location\") pod \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\" (UID: \"48de669d-e0a7-4f8b-9f02-37f4d59ddd65\") "
Sep 17 18:08:28.670660 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.670638    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48de669d-e0a7-4f8b-9f02-37f4d59ddd65" (UID: "48de669d-e0a7-4f8b-9f02-37f4d59ddd65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:28.671043 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.671020    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.671135 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.671098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48de669d-e0a7-4f8b-9f02-37f4d59ddd65" (UID: "48de669d-e0a7-4f8b-9f02-37f4d59ddd65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:28.671135 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.671100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48de669d-e0a7-4f8b-9f02-37f4d59ddd65" (UID: "48de669d-e0a7-4f8b-9f02-37f4d59ddd65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:28.672556 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.672530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48de669d-e0a7-4f8b-9f02-37f4d59ddd65" (UID: "48de669d-e0a7-4f8b-9f02-37f4d59ddd65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:28.672556 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.672549    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-api-access-9sz75" (OuterVolumeSpecName: "kube-api-access-9sz75") pod "48de669d-e0a7-4f8b-9f02-37f4d59ddd65" (UID: "48de669d-e0a7-4f8b-9f02-37f4d59ddd65"). InnerVolumeSpecName "kube-api-access-9sz75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:28.756415 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.756378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:28.772402 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.772369    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sz75\" (UniqueName: \"kubernetes.io/projected/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kube-api-access-9sz75\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.772402 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.772403    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.772603 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.772417    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.772603 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.772445    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48de669d-e0a7-4f8b-9f02-37f4d59ddd65-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.798165 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798125    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"]
Sep 17 18:08:28.798573 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798554    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container"
Sep 17 18:08:28.798661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798575    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container"
Sep 17 18:08:28.798661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798596    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="storage-initializer"
Sep 17 18:08:28.798661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798605    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="storage-initializer"
Sep 17 18:08:28.798661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798631    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kube-rbac-proxy"
Sep 17 18:08:28.798661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798644    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kube-rbac-proxy"
Sep 17 18:08:28.798940 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798736    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kserve-container"
Sep 17 18:08:28.798940 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.798749    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" containerName="kube-rbac-proxy"
Sep 17 18:08:28.801139 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.801116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:28.803463 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.803445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:08:28.803587 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.803450    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:08:28.811413 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.811390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"]
Sep 17 18:08:28.833903 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.833866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:28.888081 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.888044    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"]
Sep 17 18:08:28.890777 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.890757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:28.892962 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.892942    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:08:28.903989 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.903826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"]
Sep 17 18:08:28.949312 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.949274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:28.974212 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.974181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:28.974345 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.974229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:28.974345 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.974277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmpj\" (UniqueName: \"kubernetes.io/projected/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-api-access-mxmpj\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:28.974345 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.974322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:28.974454 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:28.974351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.050004 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.049961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:29.075100 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.075197 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.075197 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.075197 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmn6l\" (UniqueName: \"kubernetes.io/projected/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-api-access-qmn6l\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.075197 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.075359 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.075359 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmpj\" (UniqueName: \"kubernetes.io/projected/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-api-access-mxmpj\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.075460 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.075508 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d4d8fc-f357-44c1-b207-13c6b9be7e54-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.075681 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.075681 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.075957 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.075933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.076053 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.076034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.078040 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.078020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.086450 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.086427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmpj\" (UniqueName: \"kubernetes.io/projected/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-api-access-mxmpj\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-m5qcb\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.114397 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.114365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:29.176461 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.176432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.176620 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.176485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d4d8fc-f357-44c1-b207-13c6b9be7e54-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.176620 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.176525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.176620 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.176548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.176620 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.176568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmn6l\" (UniqueName: \"kubernetes.io/projected/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-api-access-qmn6l\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.177339 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.177307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.177447 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.177347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.177447 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.177357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.179195 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.179179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d4d8fc-f357-44c1-b207-13c6b9be7e54-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.186845 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.186820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmn6l\" (UniqueName: \"kubernetes.io/projected/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-api-access-qmn6l\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-gd8lc\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.205332 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.205245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:29.391983 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.391952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"]
Sep 17 18:08:29.393842 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:08:29.393816    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c9e486_b6f1_42ec_8c0a_34646ccffd4e.slice/crio-62a675efdf2b3eec22ad55e71ce7cc712aee8696508dfcd62ea031aa0acf0ad4 WatchSource:0}: Error finding container 62a675efdf2b3eec22ad55e71ce7cc712aee8696508dfcd62ea031aa0acf0ad4: Status 404 returned error can't find the container with id 62a675efdf2b3eec22ad55e71ce7cc712aee8696508dfcd62ea031aa0acf0ad4
Sep 17 18:08:29.477665 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.477631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"]
Sep 17 18:08:29.481367 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:08:29.481339    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d4d8fc_f357_44c1_b207_13c6b9be7e54.slice/crio-1eed3359d33afb0005bc783c399d57972d544fbc69c058005cee83292471376c WatchSource:0}: Error finding container 1eed3359d33afb0005bc783c399d57972d544fbc69c058005cee83292471376c: Status 404 returned error can't find the container with id 1eed3359d33afb0005bc783c399d57972d544fbc69c058005cee83292471376c
Sep 17 18:08:29.523317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.523283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4" event={"ID":"48de669d-e0a7-4f8b-9f02-37f4d59ddd65","Type":"ContainerDied","Data":"984b88247000a64d38fc2042af658ece29094f0053b8d15c18431b866b719f8b"}
Sep 17 18:08:29.523756 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.523361    2563 scope.go:117] "RemoveContainer" containerID="78a0df99c4845283831fe2218b8febdd7613db2bdc23a6ffc4187e02cd8445e9"
Sep 17 18:08:29.523756 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.523299    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"
Sep 17 18:08:29.525182 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.525154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerStarted","Data":"497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921"}
Sep 17 18:08:29.525298 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.525215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerStarted","Data":"62a675efdf2b3eec22ad55e71ce7cc712aee8696508dfcd62ea031aa0acf0ad4"}
Sep 17 18:08:29.527394 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.527364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerStarted","Data":"1eed3359d33afb0005bc783c399d57972d544fbc69c058005cee83292471376c"}
Sep 17 18:08:29.533673 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.533655    2563 scope.go:117] "RemoveContainer" containerID="485b8af7af0c55c76a4c9f13f8070e4d15750848f2915e1fbbfbdadcf9dca9d0"
Sep 17 18:08:29.554310 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.554288    2563 scope.go:117] "RemoveContainer" containerID="77f049467ac9eb4858b1f6a17a891f3f02e82da969739f6990cd4e8620eb32d2"
Sep 17 18:08:29.580453 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.580419    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"]
Sep 17 18:08:29.589260 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:29.589229    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-9ssf4"]
Sep 17 18:08:30.535187 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:30.535149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerStarted","Data":"0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3"}
Sep 17 18:08:30.959921 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:30.959890    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48de669d-e0a7-4f8b-9f02-37f4d59ddd65" path="/var/lib/kubelet/pods/48de669d-e0a7-4f8b-9f02-37f4d59ddd65/volumes"
Sep 17 18:08:33.552010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:33.551970    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerID="497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921" exitCode=0
Sep 17 18:08:33.552405 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:33.552043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerDied","Data":"497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921"}
Sep 17 18:08:33.553734 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:33.553707    2563 generic.go:356] "Generic (PLEG): container finished" podID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerID="0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3" exitCode=0
Sep 17 18:08:33.553858 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:33.553742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerDied","Data":"0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3"}
Sep 17 18:08:34.560278 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.560247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerStarted","Data":"0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d"}
Sep 17 18:08:34.560709 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.560322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerStarted","Data":"7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e"}
Sep 17 18:08:34.560709 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.560557    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:34.562536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.562511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerStarted","Data":"79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328"}
Sep 17 18:08:34.562657 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.562566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerStarted","Data":"2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7"}
Sep 17 18:08:34.562748 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.562732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:34.582971 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.582918    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podStartSLOduration=2.728107225 podStartE2EDuration="6.582901198s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.854793973s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:34.581110291 +0000 UTC m=+2000.108892115" watchObservedRunningTime="2026-09-17 18:08:34.582901198 +0000 UTC m=+2000.110683023"
Sep 17 18:08:34.599745 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:34.599698    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podStartSLOduration=2.844282462 podStartE2EDuration="6.599682927s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.755400465s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:34.597930599 +0000 UTC m=+2000.125712425" watchObservedRunningTime="2026-09-17 18:08:34.599682927 +0000 UTC m=+2000.127464756"
Sep 17 18:08:35.567966 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:35.567931    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:35.568344 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:35.567998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:35.569229 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:35.569199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:08:35.569314 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:35.569211    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:08:36.573159 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:36.573121    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:08:36.573612 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:36.573205    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:08:37.706148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:37.706099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:41.578584 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:41.578553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:08:41.579230 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:41.579192    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:08:41.579392 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:41.579308    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:08:41.579736 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:41.579712    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:08:42.839181 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:42.839096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:42.946761 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:42.946731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:51.579228 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:51.579178    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:08:51.579810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:08:51.579660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:01.579837 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:01.579765    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:09:01.579837 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:01.579765    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:11.579122 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:11.579083    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:09:11.580322 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:11.580292    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:21.579546 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:21.579505    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:09:21.579983 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:21.579641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:21.650680 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:21.650630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:24.099504 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:24.099467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:29.264546 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.264514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:29.300470 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.300439    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"]
Sep 17 18:09:29.303105 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.303087    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.305391 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.305368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 18:09:29.305523 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.305393    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:09:29.317116 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.317090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"]
Sep 17 18:09:29.424882 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.424834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.424882 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.424885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.425119 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.424969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/232831da-1289-46c3-8a39-3377ddbffb51-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.425119 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.425035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.425119 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.425066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcrx\" (UniqueName: \"kubernetes.io/projected/232831da-1289-46c3-8a39-3377ddbffb51-kube-api-access-gtcrx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.525934 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.525851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526090 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.525995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/232831da-1289-46c3-8a39-3377ddbffb51-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526090 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.526040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526090 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.526066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcrx\" (UniqueName: \"kubernetes.io/projected/232831da-1289-46c3-8a39-3377ddbffb51-kube-api-access-gtcrx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526090 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.526087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526295 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:09:29.526191    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 18:09:29.526295 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:09:29.526262    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls podName:232831da-1289-46c3-8a39-3377ddbffb51 nodeName:}" failed. No retries permitted until 2026-09-17 18:09:30.026240061 +0000 UTC m=+2055.554021872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-68btf" (UID: "232831da-1289-46c3-8a39-3377ddbffb51") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 18:09:29.526418 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.526358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/232831da-1289-46c3-8a39-3377ddbffb51-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526622 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.526603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.526688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.526671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.535025 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.534993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcrx\" (UniqueName: \"kubernetes.io/projected/232831da-1289-46c3-8a39-3377ddbffb51-kube-api-access-gtcrx\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:29.824229 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:29.824136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:30.030047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:30.029991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:30.032642 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:30.032606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-68btf\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:30.214020 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:30.213978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:30.486991 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:30.486919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"]
Sep 17 18:09:30.489925 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:09:30.489897    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232831da_1289_46c3_8a39_3377ddbffb51.slice/crio-cb9aebac73e5846e5cc9be2eb70efbfed598c1bc5680a85fa5910c426269c809 WatchSource:0}: Error finding container cb9aebac73e5846e5cc9be2eb70efbfed598c1bc5680a85fa5910c426269c809: Status 404 returned error can't find the container with id cb9aebac73e5846e5cc9be2eb70efbfed598c1bc5680a85fa5910c426269c809
Sep 17 18:09:30.839942 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:30.839834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerStarted","Data":"bc681c196e4bde6fa90eeb4b74802b7e1608a18b7c433f4a5cdca0437702d256"}
Sep 17 18:09:30.839942 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:30.839929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerStarted","Data":"cb9aebac73e5846e5cc9be2eb70efbfed598c1bc5680a85fa5910c426269c809"}
Sep 17 18:09:31.579292 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:31.579254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:09:31.579695 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:31.579673    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:35.868638 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:35.868603    2563 generic.go:356] "Generic (PLEG): container finished" podID="232831da-1289-46c3-8a39-3377ddbffb51" containerID="bc681c196e4bde6fa90eeb4b74802b7e1608a18b7c433f4a5cdca0437702d256" exitCode=0
Sep 17 18:09:35.869026 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:35.868684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerDied","Data":"bc681c196e4bde6fa90eeb4b74802b7e1608a18b7c433f4a5cdca0437702d256"}
Sep 17 18:09:39.896008 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:39.895970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerStarted","Data":"5312d99852c2fe673d2da452c8913ff236879059e7e21fa51dbbe3d2dce9206d"}
Sep 17 18:09:39.896388 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:39.896032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerStarted","Data":"ca58d863caccaa417659feaf00997b637aa06fb9bb5f11c3087c31619a72c8e8"}
Sep 17 18:09:39.896388 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:39.896314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:39.896482 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:39.896464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:39.897950 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:39.897926    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:09:39.920877 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:39.920782    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podStartSLOduration=2.492613206 podStartE2EDuration="10.920768818s" totalImagesPullingTime="3.788478441s" totalInitContainerRuntime="4.639677171s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:39.918127037 +0000 UTC m=+2065.445908863" watchObservedRunningTime="2026-09-17 18:09:39.920768818 +0000 UTC m=+2065.448550643"
Sep 17 18:09:40.901470 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:40.901422    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:09:41.578972 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:41.578928    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:09:41.580175 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:41.580142    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:42.953627 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:42.953581    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:09:45.907136 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:45.907107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:45.907750 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:45.907723    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:09:51.580966 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:51.580935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:09:52.958232 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:52.958202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:09:55.908378 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:55.908348    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:09:56.320936 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:56.320896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:58.843191 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:58.843159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"]
Sep 17 18:09:58.843601 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:58.843409    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" containerID="cri-o://2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7" gracePeriod=30
Sep 17 18:09:58.843601 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:58.843479    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kube-rbac-proxy" containerID="cri-o://79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328" gracePeriod=30
Sep 17 18:09:58.876969 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:58.876933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:58.994885 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:58.994853    2563 generic.go:356] "Generic (PLEG): container finished" podID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerID="79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328" exitCode=2
Sep 17 18:09:58.995045 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:09:58.994927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerDied","Data":"79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328"}
Sep 17 18:10:01.573407 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:01.573358    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 17 18:10:01.579772 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:01.579745    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 18:10:04.001342 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.001316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:10:04.021525 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.021232    2563 generic.go:356] "Generic (PLEG): container finished" podID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerID="2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7" exitCode=0
Sep 17 18:10:04.021525 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.021406    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"
Sep 17 18:10:04.021525 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.021437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerDied","Data":"2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7"}
Sep 17 18:10:04.021867 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.021567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc" event={"ID":"24d4d8fc-f357-44c1-b207-13c6b9be7e54","Type":"ContainerDied","Data":"1eed3359d33afb0005bc783c399d57972d544fbc69c058005cee83292471376c"}
Sep 17 18:10:04.021867 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.021617    2563 scope.go:117] "RemoveContainer" containerID="79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328"
Sep 17 18:10:04.032341 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.032313    2563 scope.go:117] "RemoveContainer" containerID="2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7"
Sep 17 18:10:04.037348 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmn6l\" (UniqueName: \"kubernetes.io/projected/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-api-access-qmn6l\") pod \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") "
Sep 17 18:10:04.037478 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037397    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d4d8fc-f357-44c1-b207-13c6b9be7e54-proxy-tls\") pod \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") "
Sep 17 18:10:04.037546 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037509    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-cabundle-cert\") pod \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") "
Sep 17 18:10:04.037546 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-rbac-proxy-sar-config\") pod \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") "
Sep 17 18:10:04.037654 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037582    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kserve-provision-location\") pod \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\" (UID: \"24d4d8fc-f357-44c1-b207-13c6b9be7e54\") "
Sep 17 18:10:04.037909 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24d4d8fc-f357-44c1-b207-13c6b9be7e54" (UID: "24d4d8fc-f357-44c1-b207-13c6b9be7e54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:04.038003 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.037986    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:04.038193 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.038167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24d4d8fc-f357-44c1-b207-13c6b9be7e54" (UID: "24d4d8fc-f357-44c1-b207-13c6b9be7e54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:04.038406 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.038361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24d4d8fc-f357-44c1-b207-13c6b9be7e54" (UID: "24d4d8fc-f357-44c1-b207-13c6b9be7e54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:04.039683 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.039642    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d4d8fc-f357-44c1-b207-13c6b9be7e54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24d4d8fc-f357-44c1-b207-13c6b9be7e54" (UID: "24d4d8fc-f357-44c1-b207-13c6b9be7e54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:04.039989 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.039953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-api-access-qmn6l" (OuterVolumeSpecName: "kube-api-access-qmn6l") pod "24d4d8fc-f357-44c1-b207-13c6b9be7e54" (UID: "24d4d8fc-f357-44c1-b207-13c6b9be7e54"). InnerVolumeSpecName "kube-api-access-qmn6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:04.041486 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.041467    2563 scope.go:117] "RemoveContainer" containerID="0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3"
Sep 17 18:10:04.056746 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.056721    2563 scope.go:117] "RemoveContainer" containerID="79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328"
Sep 17 18:10:04.057107 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:10:04.057088    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328\": container with ID starting with 79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328 not found: ID does not exist" containerID="79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328"
Sep 17 18:10:04.057175 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.057118    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328"} err="failed to get container status \"79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328\": rpc error: code = NotFound desc = could not find container \"79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328\": container with ID starting with 79e063e5eb818caa74a550942095dd76e6bc6393cf69b0799154ef8eda4be328 not found: ID does not exist"
Sep 17 18:10:04.057175 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.057141    2563 scope.go:117] "RemoveContainer" containerID="2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7"
Sep 17 18:10:04.057430 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:10:04.057413    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7\": container with ID starting with 2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7 not found: ID does not exist" containerID="2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7"
Sep 17 18:10:04.057483 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.057433    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7"} err="failed to get container status \"2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7\": rpc error: code = NotFound desc = could not find container \"2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7\": container with ID starting with 2afd55f9437a96f187549ba2e08c356d2e9f556df7a80cd8951c6dc6c5077fe7 not found: ID does not exist"
Sep 17 18:10:04.057483 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.057449    2563 scope.go:117] "RemoveContainer" containerID="0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3"
Sep 17 18:10:04.057744 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:10:04.057726    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3\": container with ID starting with 0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3 not found: ID does not exist" containerID="0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3"
Sep 17 18:10:04.057843 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.057746    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3"} err="failed to get container status \"0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3\": rpc error: code = NotFound desc = could not find container \"0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3\": container with ID starting with 0eb81982ff202842416b69af35302a61044ca81a9b52f63a185e64bcd5b368f3 not found: ID does not exist"
Sep 17 18:10:04.139284 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.139182    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:04.139284 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.139215    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:04.139284 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.139227    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmn6l\" (UniqueName: \"kubernetes.io/projected/24d4d8fc-f357-44c1-b207-13c6b9be7e54-kube-api-access-qmn6l\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:04.139284 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.139236    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24d4d8fc-f357-44c1-b207-13c6b9be7e54-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:04.344904 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.344846    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"]
Sep 17 18:10:04.348349 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.348320    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-gd8lc"]
Sep 17 18:10:04.959080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:04.959048    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" path="/var/lib/kubelet/pods/24d4d8fc-f357-44c1-b207-13c6b9be7e54/volumes"
Sep 17 18:10:10.905416 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:10.905385    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"]
Sep 17 18:10:10.905856 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:10.905650    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container" containerID="cri-o://ca58d863caccaa417659feaf00997b637aa06fb9bb5f11c3087c31619a72c8e8" gracePeriod=30
Sep 17 18:10:10.905856 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:10.905691    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" containerID="cri-o://5312d99852c2fe673d2da452c8913ff236879059e7e21fa51dbbe3d2dce9206d" gracePeriod=30
Sep 17 18:10:10.949913 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:10.949877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:11.058584 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:11.058553    2563 generic.go:356] "Generic (PLEG): container finished" podID="232831da-1289-46c3-8a39-3377ddbffb51" containerID="5312d99852c2fe673d2da452c8913ff236879059e7e21fa51dbbe3d2dce9206d" exitCode=2
Sep 17 18:10:11.058755 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:11.058612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerDied","Data":"5312d99852c2fe673d2da452c8913ff236879059e7e21fa51dbbe3d2dce9206d"}
Sep 17 18:10:14.305514 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:14.305479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:14.374781 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:14.374748    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:15.901683 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:15.901641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 18:10:16.025662 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:16.025613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:16.238912 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:16.238877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:20.902637 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:20.902593    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 18:10:25.902212 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:25.902168    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 18:10:25.902647 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:25.902310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:10:30.905046 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:30.904998    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 18:10:33.298374 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:33.298268    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:10:33.303391 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:33.303359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:10:35.901843 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:35.901775    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 18:10:40.902174 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:40.902129    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 18:10:41.204405 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.204370    2563 generic.go:356] "Generic (PLEG): container finished" podID="232831da-1289-46c3-8a39-3377ddbffb51" containerID="ca58d863caccaa417659feaf00997b637aa06fb9bb5f11c3087c31619a72c8e8" exitCode=137
Sep 17 18:10:41.204682 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.204423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerDied","Data":"ca58d863caccaa417659feaf00997b637aa06fb9bb5f11c3087c31619a72c8e8"}
Sep 17 18:10:41.563281 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.563253    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:10:41.677232 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-kube-rbac-proxy-sar-config\") pod \"232831da-1289-46c3-8a39-3377ddbffb51\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") "
Sep 17 18:10:41.677426 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls\") pod \"232831da-1289-46c3-8a39-3377ddbffb51\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") "
Sep 17 18:10:41.677426 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/232831da-1289-46c3-8a39-3377ddbffb51-kserve-provision-location\") pod \"232831da-1289-46c3-8a39-3377ddbffb51\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") "
Sep 17 18:10:41.677426 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-cabundle-cert\") pod \"232831da-1289-46c3-8a39-3377ddbffb51\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") "
Sep 17 18:10:41.677426 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtcrx\" (UniqueName: \"kubernetes.io/projected/232831da-1289-46c3-8a39-3377ddbffb51-kube-api-access-gtcrx\") pod \"232831da-1289-46c3-8a39-3377ddbffb51\" (UID: \"232831da-1289-46c3-8a39-3377ddbffb51\") "
Sep 17 18:10:41.677760 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "232831da-1289-46c3-8a39-3377ddbffb51" (UID: "232831da-1289-46c3-8a39-3377ddbffb51"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:41.677855 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.677822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "232831da-1289-46c3-8a39-3377ddbffb51" (UID: "232831da-1289-46c3-8a39-3377ddbffb51"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:41.679485 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.679462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232831da-1289-46c3-8a39-3377ddbffb51-kube-api-access-gtcrx" (OuterVolumeSpecName: "kube-api-access-gtcrx") pod "232831da-1289-46c3-8a39-3377ddbffb51" (UID: "232831da-1289-46c3-8a39-3377ddbffb51"). InnerVolumeSpecName "kube-api-access-gtcrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:41.679607 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.679563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "232831da-1289-46c3-8a39-3377ddbffb51" (UID: "232831da-1289-46c3-8a39-3377ddbffb51"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:41.688855 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.688830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232831da-1289-46c3-8a39-3377ddbffb51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "232831da-1289-46c3-8a39-3377ddbffb51" (UID: "232831da-1289-46c3-8a39-3377ddbffb51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:41.779111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.779034    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/232831da-1289-46c3-8a39-3377ddbffb51-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:41.779111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.779060    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/232831da-1289-46c3-8a39-3377ddbffb51-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:41.779111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.779070    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:41.779111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.779079    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtcrx\" (UniqueName: \"kubernetes.io/projected/232831da-1289-46c3-8a39-3377ddbffb51-kube-api-access-gtcrx\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:41.779111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:41.779088    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/232831da-1289-46c3-8a39-3377ddbffb51-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:10:42.211338 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.211293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf" event={"ID":"232831da-1289-46c3-8a39-3377ddbffb51","Type":"ContainerDied","Data":"cb9aebac73e5846e5cc9be2eb70efbfed598c1bc5680a85fa5910c426269c809"}
Sep 17 18:10:42.211755 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.211376    2563 scope.go:117] "RemoveContainer" containerID="5312d99852c2fe673d2da452c8913ff236879059e7e21fa51dbbe3d2dce9206d"
Sep 17 18:10:42.211755 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.211309    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"
Sep 17 18:10:42.219664 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.219645    2563 scope.go:117] "RemoveContainer" containerID="ca58d863caccaa417659feaf00997b637aa06fb9bb5f11c3087c31619a72c8e8"
Sep 17 18:10:42.227300 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.227274    2563 scope.go:117] "RemoveContainer" containerID="bc681c196e4bde6fa90eeb4b74802b7e1608a18b7c433f4a5cdca0437702d256"
Sep 17 18:10:42.235334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.235313    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"]
Sep 17 18:10:42.239273 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.239252    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-68btf"]
Sep 17 18:10:42.958434 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:10:42.958402    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232831da-1289-46c3-8a39-3377ddbffb51" path="/var/lib/kubelet/pods/232831da-1289-46c3-8a39-3377ddbffb51/volumes"
Sep 17 18:11:31.166727 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:31.166681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:49.153321 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:49.153288    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"]
Sep 17 18:11:49.153910 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:49.153604    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" containerID="cri-o://7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e" gracePeriod=30
Sep 17 18:11:49.153910 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:49.153672    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kube-rbac-proxy" containerID="cri-o://0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d" gracePeriod=30
Sep 17 18:11:49.189333 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:49.189307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:49.530692 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:49.530659    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerID="0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d" exitCode=2
Sep 17 18:11:49.530888 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:49.530726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerDied","Data":"0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d"}
Sep 17 18:11:51.574096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:51.574053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 17 18:11:52.953984 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:52.953941    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:11:54.109534 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.109503    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:11:54.213594 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.213501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:54.218347 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-cabundle-cert\") pod \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") "
Sep 17 18:11:54.218496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-rbac-proxy-sar-config\") pod \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") "
Sep 17 18:11:54.218496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kserve-provision-location\") pod \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") "
Sep 17 18:11:54.218496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxmpj\" (UniqueName: \"kubernetes.io/projected/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-api-access-mxmpj\") pod \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") "
Sep 17 18:11:54.218664 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-proxy-tls\") pod \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\" (UID: \"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e\") "
Sep 17 18:11:54.218737 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218703    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" (UID: "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:54.218818 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218762    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" (UID: "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:11:54.218882 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218870    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.218939 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.218889    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.219078 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.219057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" (UID: "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:54.220651 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.220622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-api-access-mxmpj" (OuterVolumeSpecName: "kube-api-access-mxmpj") pod "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" (UID: "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e"). InnerVolumeSpecName "kube-api-access-mxmpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:54.220996 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.220975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" (UID: "b9c9e486-b6f1-42ec-8c0a-34646ccffd4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:54.280748 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.280721    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"]
Sep 17 18:11:54.281132 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281117    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281135    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281145    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281151    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281176    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281184    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281194    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="storage-initializer"
Sep 17 18:11:54.281198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281200    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="storage-initializer"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281208    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281219    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281224    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281229    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281234    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281243    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="storage-initializer"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281248    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="storage-initializer"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281257    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="storage-initializer"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281267    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="storage-initializer"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281321    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kserve-container"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281328    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281335    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281344    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="232831da-1289-46c3-8a39-3377ddbffb51" containerName="kserve-container"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281351    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d4d8fc-f357-44c1-b207-13c6b9be7e54" containerName="kube-rbac-proxy"
Sep 17 18:11:54.281444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.281357    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerName="kserve-container"
Sep 17 18:11:54.284829 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.284812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.287265 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.287244    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 18:11:54.287379 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.287272    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:11:54.295013 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.294988    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"]
Sep 17 18:11:54.299240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.299219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"]
Sep 17 18:11:54.299343 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.299331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.300569 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.300509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:54.301834 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.301813    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:11:54.313493 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.313414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"]
Sep 17 18:11:54.319991 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.319972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.320096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjjk\" (UniqueName: \"kubernetes.io/projected/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-api-access-jdjjk\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.320161 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.320217 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.320272 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.320331 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320289    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.320331 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320306    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmpj\" (UniqueName: \"kubernetes.io/projected/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-kube-api-access-mxmpj\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.320331 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.320320    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:11:54.396609 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.396574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:54.421642 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.421767 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.421767 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc8a582-642c-4d6c-95a3-4630537a8461-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.421767 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.421922 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.421922 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.421922 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc8a582-642c-4d6c-95a3-4630537a8461-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.422075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrgr\" (UniqueName: \"kubernetes.io/projected/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-api-access-pjrgr\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.422075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjjk\" (UniqueName: \"kubernetes.io/projected/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-api-access-jdjjk\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.422075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.421999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.422075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.422006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.422880 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.422852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.423048 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.423026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.425342 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.425314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.431286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.431264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjjk\" (UniqueName: \"kubernetes.io/projected/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-api-access-jdjjk\") pod \"isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.457165 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.457141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:54.523456 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.523376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.523456 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.523442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc8a582-642c-4d6c-95a3-4630537a8461-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.523625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.523491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc8a582-642c-4d6c-95a3-4630537a8461-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.523625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.523507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrgr\" (UniqueName: \"kubernetes.io/projected/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-api-access-pjrgr\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.523625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.523540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.523968 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.523945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc8a582-642c-4d6c-95a3-4630537a8461-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.524154 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.524137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.524218 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.524202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.525908 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.525892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc8a582-642c-4d6c-95a3-4630537a8461-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.531991 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.531971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrgr\" (UniqueName: \"kubernetes.io/projected/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-api-access-pjrgr\") pod \"isvc-canary-rollback-predictor-786544c6c8-266bn\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.555733 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.555707    2563 generic.go:356] "Generic (PLEG): container finished" podID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" containerID="7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e" exitCode=0
Sep 17 18:11:54.555863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.555743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerDied","Data":"7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e"}
Sep 17 18:11:54.555863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.555770    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"
Sep 17 18:11:54.555863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.555827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb" event={"ID":"b9c9e486-b6f1-42ec-8c0a-34646ccffd4e","Type":"ContainerDied","Data":"62a675efdf2b3eec22ad55e71ce7cc712aee8696508dfcd62ea031aa0acf0ad4"}
Sep 17 18:11:54.555863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.555855    2563 scope.go:117] "RemoveContainer" containerID="0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d"
Sep 17 18:11:54.564957 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.564942    2563 scope.go:117] "RemoveContainer" containerID="7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e"
Sep 17 18:11:54.572191 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.572176    2563 scope.go:117] "RemoveContainer" containerID="497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921"
Sep 17 18:11:54.577205 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.577185    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"]
Sep 17 18:11:54.580217 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.580200    2563 scope.go:117] "RemoveContainer" containerID="0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d"
Sep 17 18:11:54.580501 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:11:54.580477    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d\": container with ID starting with 0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d not found: ID does not exist" containerID="0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d"
Sep 17 18:11:54.580561 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.580511    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d"} err="failed to get container status \"0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d\": rpc error: code = NotFound desc = could not find container \"0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d\": container with ID starting with 0a36f9b23512aad6a806730a546ed2fea4b3db5487050567ad08db5086d1eb3d not found: ID does not exist"
Sep 17 18:11:54.580561 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.580533    2563 scope.go:117] "RemoveContainer" containerID="7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e"
Sep 17 18:11:54.580824 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:11:54.580784    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e\": container with ID starting with 7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e not found: ID does not exist" containerID="7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e"
Sep 17 18:11:54.580869 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.580830    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e"} err="failed to get container status \"7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e\": rpc error: code = NotFound desc = could not find container \"7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e\": container with ID starting with 7749d1f39a0f5438bde3d3dfd0796be0b41fb45c4d8b1ec9414834864e54654e not found: ID does not exist"
Sep 17 18:11:54.580869 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.580847    2563 scope.go:117] "RemoveContainer" containerID="497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921"
Sep 17 18:11:54.581097 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:11:54.581077    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921\": container with ID starting with 497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921 not found: ID does not exist" containerID="497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921"
Sep 17 18:11:54.581155 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.581102    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921"} err="failed to get container status \"497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921\": rpc error: code = NotFound desc = could not find container \"497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921\": container with ID starting with 497fbf4e965ee7406776d68dee88bfd511ad17ae6eed3bcd4670a381f4b82921 not found: ID does not exist"
Sep 17 18:11:54.581197 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.581184    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-m5qcb"]
Sep 17 18:11:54.596515 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.596498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:11:54.612661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.612637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:11:54.875884 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.875860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"]
Sep 17 18:11:54.878308 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:11:54.878274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa45b34_2f6c_4b29_b104_c1b8b58d2036.slice/crio-57d9f122c488bba92408cddc9a14204ca9a4f0d18036db18d3a562c9938226af WatchSource:0}: Error finding container 57d9f122c488bba92408cddc9a14204ca9a4f0d18036db18d3a562c9938226af: Status 404 returned error can't find the container with id 57d9f122c488bba92408cddc9a14204ca9a4f0d18036db18d3a562c9938226af
Sep 17 18:11:54.895071 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.895051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"]
Sep 17 18:11:54.897203 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:11:54.897176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc8a582_642c_4d6c_95a3_4630537a8461.slice/crio-27292e20e64430cb88720a0c25d4769f24ba73d1166a2e26209ed98ae6624623 WatchSource:0}: Error finding container 27292e20e64430cb88720a0c25d4769f24ba73d1166a2e26209ed98ae6624623: Status 404 returned error can't find the container with id 27292e20e64430cb88720a0c25d4769f24ba73d1166a2e26209ed98ae6624623
Sep 17 18:11:54.965092 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:54.965052    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c9e486-b6f1-42ec-8c0a-34646ccffd4e" path="/var/lib/kubelet/pods/b9c9e486-b6f1-42ec-8c0a-34646ccffd4e/volumes"
Sep 17 18:11:55.563310 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:55.563271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerStarted","Data":"c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490"}
Sep 17 18:11:55.563818 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:55.563348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerStarted","Data":"57d9f122c488bba92408cddc9a14204ca9a4f0d18036db18d3a562c9938226af"}
Sep 17 18:11:55.565274 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:55.565237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerStarted","Data":"28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a"}
Sep 17 18:11:55.565407 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:55.565303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerStarted","Data":"27292e20e64430cb88720a0c25d4769f24ba73d1166a2e26209ed98ae6624623"}
Sep 17 18:11:59.586161 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:59.586121    2563 generic.go:356] "Generic (PLEG): container finished" podID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerID="c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490" exitCode=0
Sep 17 18:11:59.586621 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:59.586198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerDied","Data":"c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490"}
Sep 17 18:11:59.587902 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:59.587881    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerID="28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a" exitCode=0
Sep 17 18:11:59.587993 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:11:59.587941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerDied","Data":"28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a"}
Sep 17 18:12:00.595436 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.595401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerStarted","Data":"0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc"}
Sep 17 18:12:00.595857 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.595474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerStarted","Data":"a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8"}
Sep 17 18:12:00.595857 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.595694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:12:00.597653 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.597629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerStarted","Data":"5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c"}
Sep 17 18:12:00.597774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.597678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerStarted","Data":"80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8"}
Sep 17 18:12:00.597935 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.597919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:12:00.597993 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.597974    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:12:00.599464 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.599441    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:00.614478 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.614427    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podStartSLOduration=2.830405287 podStartE2EDuration="6.614408381s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784003094s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:00.614303385 +0000 UTC m=+2206.142085212" watchObservedRunningTime="2026-09-17 18:12:00.614408381 +0000 UTC m=+2206.142190208"
Sep 17 18:12:00.636396 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:00.636344    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podStartSLOduration=2.694266373 podStartE2EDuration="6.636325933s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.94205956s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:00.63297576 +0000 UTC m=+2206.160757625" watchObservedRunningTime="2026-09-17 18:12:00.636325933 +0000 UTC m=+2206.164107760"
Sep 17 18:12:01.602710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:01.602656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:01.603239 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:01.602935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:12:01.604353 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:01.604328    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:02.607615 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:02.607576    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:06.607381 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:06.607352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:12:06.607948 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:06.607918    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:07.612281 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:07.612254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:12:07.612768 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:07.612742    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:16.608864 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:16.608814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:17.613166 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:17.613119    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:26.608273 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:26.608227    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:27.613338 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:27.613299    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:28.394398 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:28.394356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:33.573741 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:33.573697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:33.966152 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:33.966119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:36.608383 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:36.608339    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:37.613484 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:37.613396    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:46.608109 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:46.608063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:47.613120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:47.613073    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:12:56.608342 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:56.608298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:12:57.613415 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:12:57.613372    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:13:06.608064 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:06.608023    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:13:07.612851 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:07.612804    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:13:16.608958 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.608928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:13:16.733859 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.733826    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"]
Sep 17 18:13:16.734111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.734059    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container" containerID="cri-o://a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8" gracePeriod=30
Sep 17 18:13:16.734249 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.734138    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kube-rbac-proxy" containerID="cri-o://0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc" gracePeriod=30
Sep 17 18:13:16.735416 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.735397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:13:16.964811 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.964768    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerID="0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc" exitCode=2
Sep 17 18:13:16.964995 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:16.964823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerDied","Data":"0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc"}
Sep 17 18:13:17.607815 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:17.607755    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 17 18:13:21.584671 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.584645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:13:21.709004 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.708921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc8a582-642c-4d6c-95a3-4630537a8461-proxy-tls\") pod \"fcc8a582-642c-4d6c-95a3-4630537a8461\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") "
Sep 17 18:13:21.709004 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.708977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-cabundle-cert\") pod \"fcc8a582-642c-4d6c-95a3-4630537a8461\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") "
Sep 17 18:13:21.709202 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.709007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc8a582-642c-4d6c-95a3-4630537a8461-kserve-provision-location\") pod \"fcc8a582-642c-4d6c-95a3-4630537a8461\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") "
Sep 17 18:13:21.709202 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.709038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-rbac-proxy-sar-config\") pod \"fcc8a582-642c-4d6c-95a3-4630537a8461\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") "
Sep 17 18:13:21.709202 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.709064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjrgr\" (UniqueName: \"kubernetes.io/projected/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-api-access-pjrgr\") pod \"fcc8a582-642c-4d6c-95a3-4630537a8461\" (UID: \"fcc8a582-642c-4d6c-95a3-4630537a8461\") "
Sep 17 18:13:21.709411 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.709373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcc8a582-642c-4d6c-95a3-4630537a8461-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcc8a582-642c-4d6c-95a3-4630537a8461" (UID: "fcc8a582-642c-4d6c-95a3-4630537a8461"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:21.709553 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.709415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcc8a582-642c-4d6c-95a3-4630537a8461" (UID: "fcc8a582-642c-4d6c-95a3-4630537a8461"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:21.709553 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.709420    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcc8a582-642c-4d6c-95a3-4630537a8461" (UID: "fcc8a582-642c-4d6c-95a3-4630537a8461"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:21.711229 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.711202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-api-access-pjrgr" (OuterVolumeSpecName: "kube-api-access-pjrgr") pod "fcc8a582-642c-4d6c-95a3-4630537a8461" (UID: "fcc8a582-642c-4d6c-95a3-4630537a8461"). InnerVolumeSpecName "kube-api-access-pjrgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:21.711229 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.711223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc8a582-642c-4d6c-95a3-4630537a8461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcc8a582-642c-4d6c-95a3-4630537a8461" (UID: "fcc8a582-642c-4d6c-95a3-4630537a8461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:21.810120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.810086    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcc8a582-642c-4d6c-95a3-4630537a8461-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:21.810120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.810116    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:21.810120 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.810128    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcc8a582-642c-4d6c-95a3-4630537a8461-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:21.810384 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.810137    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:21.810384 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.810146    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjrgr\" (UniqueName: \"kubernetes.io/projected/fcc8a582-642c-4d6c-95a3-4630537a8461-kube-api-access-pjrgr\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:21.992250 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.992171    2563 generic.go:356] "Generic (PLEG): container finished" podID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerID="a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8" exitCode=0
Sep 17 18:13:21.992250 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.992215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerDied","Data":"a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8"}
Sep 17 18:13:21.992250 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.992222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"
Sep 17 18:13:21.992557 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.992269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn" event={"ID":"fcc8a582-642c-4d6c-95a3-4630537a8461","Type":"ContainerDied","Data":"27292e20e64430cb88720a0c25d4769f24ba73d1166a2e26209ed98ae6624623"}
Sep 17 18:13:21.992557 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:21.992297    2563 scope.go:117] "RemoveContainer" containerID="0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc"
Sep 17 18:13:22.001817 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.001775    2563 scope.go:117] "RemoveContainer" containerID="a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8"
Sep 17 18:13:22.009862 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.009844    2563 scope.go:117] "RemoveContainer" containerID="28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a"
Sep 17 18:13:22.017003 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.016980    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"]
Sep 17 18:13:22.017764 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.017750    2563 scope.go:117] "RemoveContainer" containerID="0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc"
Sep 17 18:13:22.018042 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:13:22.018024    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc\": container with ID starting with 0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc not found: ID does not exist" containerID="0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc"
Sep 17 18:13:22.018088 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.018054    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc"} err="failed to get container status \"0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc\": rpc error: code = NotFound desc = could not find container \"0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc\": container with ID starting with 0664f19a9a64fe55610c34508315d22972546fc14468d56bc8eebb9412abc3fc not found: ID does not exist"
Sep 17 18:13:22.018088 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.018071    2563 scope.go:117] "RemoveContainer" containerID="a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8"
Sep 17 18:13:22.018316 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:13:22.018302    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8\": container with ID starting with a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8 not found: ID does not exist" containerID="a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8"
Sep 17 18:13:22.018366 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.018319    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8"} err="failed to get container status \"a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8\": rpc error: code = NotFound desc = could not find container \"a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8\": container with ID starting with a447442a84b310e9f0394175443befaaa8e0f00f57497fb953c1b76dd892d7c8 not found: ID does not exist"
Sep 17 18:13:22.018366 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.018331    2563 scope.go:117] "RemoveContainer" containerID="28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a"
Sep 17 18:13:22.018569 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:13:22.018551    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a\": container with ID starting with 28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a not found: ID does not exist" containerID="28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a"
Sep 17 18:13:22.018624 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.018574    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a"} err="failed to get container status \"28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a\": rpc error: code = NotFound desc = could not find container \"28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a\": container with ID starting with 28d4868fb817e4204ac50794c3bcaf8c9ede24419ebc01c933d82b9ff61c545a not found: ID does not exist"
Sep 17 18:13:22.022917 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.022898    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-266bn"]
Sep 17 18:13:22.958473 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:22.958439    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" path="/var/lib/kubelet/pods/fcc8a582-642c-4d6c-95a3-4630537a8461/volumes"
Sep 17 18:13:44.297689 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.297655    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"]
Sep 17 18:13:44.298160 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.297936    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" containerID="cri-o://80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8" gracePeriod=30
Sep 17 18:13:44.298160 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.297978    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kube-rbac-proxy" containerID="cri-o://5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c" gracePeriod=30
Sep 17 18:13:44.328227 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.328199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:44.343701 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.343663    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"]
Sep 17 18:13:44.344131 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344113    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="storage-initializer"
Sep 17 18:13:44.344222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344134    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="storage-initializer"
Sep 17 18:13:44.344222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344147    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container"
Sep 17 18:13:44.344222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container"
Sep 17 18:13:44.344222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344191    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kube-rbac-proxy"
Sep 17 18:13:44.344222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344200    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kube-rbac-proxy"
Sep 17 18:13:44.344484 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kserve-container"
Sep 17 18:13:44.344484 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.344304    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc8a582-642c-4d6c-95a3-4630537a8461" containerName="kube-rbac-proxy"
Sep 17 18:13:44.347995 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.347972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.350154 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.350137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:13:44.359413 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.359388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"]
Sep 17 18:13:44.518863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.518827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4hn\" (UniqueName: \"kubernetes.io/projected/90403477-a318-4382-b20c-b5c5c70c102e-kube-api-access-mg4hn\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.519035 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.518903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.519035 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.518934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90403477-a318-4382-b20c-b5c5c70c102e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.519035 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.518957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.519035 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.519023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90403477-a318-4382-b20c-b5c5c70c102e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620056 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.619958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4hn\" (UniqueName: \"kubernetes.io/projected/90403477-a318-4382-b20c-b5c5c70c102e-kube-api-access-mg4hn\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620056 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620297 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90403477-a318-4382-b20c-b5c5c70c102e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620297 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620297 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90403477-a318-4382-b20c-b5c5c70c102e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620606 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90403477-a318-4382-b20c-b5c5c70c102e-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620829 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.620885 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.620788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.622490 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.622470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90403477-a318-4382-b20c-b5c5c70c102e-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.628271 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.628247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4hn\" (UniqueName: \"kubernetes.io/projected/90403477-a318-4382-b20c-b5c5c70c102e-kube-api-access-mg4hn\") pod \"isvc-canary-rollback-predictor-786544c6c8-wgwrn\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.658329 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.658295    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:44.931517 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:44.931492    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"]
Sep 17 18:13:44.933878 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:13:44.933851    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90403477_a318_4382_b20c_b5c5c70c102e.slice/crio-8f3e9f9184944f8b5f74b91d737c3982d205e2525967a42ef25dadd84b8ef26c WatchSource:0}: Error finding container 8f3e9f9184944f8b5f74b91d737c3982d205e2525967a42ef25dadd84b8ef26c: Status 404 returned error can't find the container with id 8f3e9f9184944f8b5f74b91d737c3982d205e2525967a42ef25dadd84b8ef26c
Sep 17 18:13:45.103252 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:45.103217    2563 generic.go:356] "Generic (PLEG): container finished" podID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerID="5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c" exitCode=2
Sep 17 18:13:45.103437 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:45.103287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerDied","Data":"5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c"}
Sep 17 18:13:45.104996 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:45.104972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerStarted","Data":"a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683"}
Sep 17 18:13:45.105111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:45.105020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerStarted","Data":"8f3e9f9184944f8b5f74b91d737c3982d205e2525967a42ef25dadd84b8ef26c"}
Sep 17 18:13:46.603569 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:46.603523    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 17 18:13:46.607916 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:46.607890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 18:13:49.131855 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.131738    2563 generic.go:356] "Generic (PLEG): container finished" podID="90403477-a318-4382-b20c-b5c5c70c102e" containerID="a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683" exitCode=0
Sep 17 18:13:49.131855 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.131814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerDied","Data":"a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683"}
Sep 17 18:13:49.343843 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.343813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:13:49.465101 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-cabundle-cert\") pod \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") "
Sep 17 18:13:49.465302 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-rbac-proxy-sar-config\") pod \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") "
Sep 17 18:13:49.465302 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdjjk\" (UniqueName: \"kubernetes.io/projected/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-api-access-jdjjk\") pod \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") "
Sep 17 18:13:49.465302 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kserve-provision-location\") pod \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") "
Sep 17 18:13:49.465302 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-proxy-tls\") pod \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\" (UID: \"eaa45b34-2f6c-4b29-b104-c1b8b58d2036\") "
Sep 17 18:13:49.465512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eaa45b34-2f6c-4b29-b104-c1b8b58d2036" (UID: "eaa45b34-2f6c-4b29-b104-c1b8b58d2036"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:49.465512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eaa45b34-2f6c-4b29-b104-c1b8b58d2036" (UID: "eaa45b34-2f6c-4b29-b104-c1b8b58d2036"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:49.465599 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eaa45b34-2f6c-4b29-b104-c1b8b58d2036" (UID: "eaa45b34-2f6c-4b29-b104-c1b8b58d2036"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:49.465682 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465656    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:49.465816 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.465686    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:49.467333 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.467310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eaa45b34-2f6c-4b29-b104-c1b8b58d2036" (UID: "eaa45b34-2f6c-4b29-b104-c1b8b58d2036"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:49.467402 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.467328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-api-access-jdjjk" (OuterVolumeSpecName: "kube-api-access-jdjjk") pod "eaa45b34-2f6c-4b29-b104-c1b8b58d2036" (UID: "eaa45b34-2f6c-4b29-b104-c1b8b58d2036"). InnerVolumeSpecName "kube-api-access-jdjjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:49.566947 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.566906    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdjjk\" (UniqueName: \"kubernetes.io/projected/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kube-api-access-jdjjk\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:49.566947 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.566941    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:49.567147 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:49.566956    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaa45b34-2f6c-4b29-b104-c1b8b58d2036-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:13:50.139479 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.139381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerStarted","Data":"f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e"}
Sep 17 18:13:50.139479 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.139462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerStarted","Data":"3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591"}
Sep 17 18:13:50.139952 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.139658    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:50.141569 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.141545    2563 generic.go:356] "Generic (PLEG): container finished" podID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerID="80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8" exitCode=0
Sep 17 18:13:50.141710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.141601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerDied","Data":"80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8"}
Sep 17 18:13:50.141710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.141615    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"
Sep 17 18:13:50.141710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.141655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w" event={"ID":"eaa45b34-2f6c-4b29-b104-c1b8b58d2036","Type":"ContainerDied","Data":"57d9f122c488bba92408cddc9a14204ca9a4f0d18036db18d3a562c9938226af"}
Sep 17 18:13:50.141710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.141694    2563 scope.go:117] "RemoveContainer" containerID="5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c"
Sep 17 18:13:50.151245 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.151217    2563 scope.go:117] "RemoveContainer" containerID="80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8"
Sep 17 18:13:50.158630 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.158612    2563 scope.go:117] "RemoveContainer" containerID="c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490"
Sep 17 18:13:50.163511 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.163472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podStartSLOduration=2.354923917 podStartE2EDuration="6.163458565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.808534648s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:50.161339205 +0000 UTC m=+2315.689121031" watchObservedRunningTime="2026-09-17 18:13:50.163458565 +0000 UTC m=+2315.691240391"
Sep 17 18:13:50.166553 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.166539    2563 scope.go:117] "RemoveContainer" containerID="5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c"
Sep 17 18:13:50.166780 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:13:50.166764    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c\": container with ID starting with 5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c not found: ID does not exist" containerID="5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c"
Sep 17 18:13:50.166850 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.166785    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c"} err="failed to get container status \"5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c\": rpc error: code = NotFound desc = could not find container \"5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c\": container with ID starting with 5212c883d30087c00a8c206b6d9775baf14e499cf1cc78cfdf80df711b6bd26c not found: ID does not exist"
Sep 17 18:13:50.166850 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.166823    2563 scope.go:117] "RemoveContainer" containerID="80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8"
Sep 17 18:13:50.167072 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:13:50.167055    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8\": container with ID starting with 80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8 not found: ID does not exist" containerID="80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8"
Sep 17 18:13:50.167114 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.167075    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8"} err="failed to get container status \"80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8\": rpc error: code = NotFound desc = could not find container \"80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8\": container with ID starting with 80615e28a573e27d83cd82fce3bc3e400fa36bf63b304d5b87dbaf34875c5ed8 not found: ID does not exist"
Sep 17 18:13:50.167114 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.167094    2563 scope.go:117] "RemoveContainer" containerID="c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490"
Sep 17 18:13:50.167315 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:13:50.167302    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490\": container with ID starting with c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490 not found: ID does not exist" containerID="c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490"
Sep 17 18:13:50.167356 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.167317    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490"} err="failed to get container status \"c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490\": rpc error: code = NotFound desc = could not find container \"c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490\": container with ID starting with c22eb38a033d88a7f1c1e0784755687d8451eb588ca347e23c2583d344099490 not found: ID does not exist"
Sep 17 18:13:50.177804 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.177777    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"]
Sep 17 18:13:50.184479 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.184457    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-76cbf6b8b7-xr99w"]
Sep 17 18:13:50.958812 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:50.958760    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" path="/var/lib/kubelet/pods/eaa45b34-2f6c-4b29-b104-c1b8b58d2036/volumes"
Sep 17 18:13:51.147969 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:51.147939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:51.149532 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:51.149503    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:13:52.153482 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:52.153447    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:13:57.158969 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:57.158939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:13:57.159491 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:13:57.159434    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:14:07.160417 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:07.160323    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:14:17.159781 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:17.159734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:14:24.148622 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:24.148582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:27.159842 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:27.159782    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:14:29.261905 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.261864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:29.304021 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.303989    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"]
Sep 17 18:14:29.304475 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304449    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="storage-initializer"
Sep 17 18:14:29.304475 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304473    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="storage-initializer"
Sep 17 18:14:29.304679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304498    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kube-rbac-proxy"
Sep 17 18:14:29.304679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304504    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kube-rbac-proxy"
Sep 17 18:14:29.304679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container"
Sep 17 18:14:29.304679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304519    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container"
Sep 17 18:14:29.304679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304580    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kserve-container"
Sep 17 18:14:29.304679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.304589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa45b34-2f6c-4b29-b104-c1b8b58d2036" containerName="kube-rbac-proxy"
Sep 17 18:14:29.308082 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.308064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.310431 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.310411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:14:29.310684 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.310668    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:14:29.319733 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.319710    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"]
Sep 17 18:14:29.420602 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.420564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53c07399-ba10-4375-a38e-3d50694d1cd3-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.420776 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.420629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.420776 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.420705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzw2\" (UniqueName: \"kubernetes.io/projected/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-api-access-kvzw2\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.420879 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.420784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.420879 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.420835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.521506 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.521420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53c07399-ba10-4375-a38e-3d50694d1cd3-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.521506 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.521483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.521740 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.521514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzw2\" (UniqueName: \"kubernetes.io/projected/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-api-access-kvzw2\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.521740 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.521564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.521740 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.521588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.521740 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:14:29.521686    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:14:29.521948 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:14:29.521771    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls podName:53c07399-ba10-4375-a38e-3d50694d1cd3 nodeName:}" failed. No retries permitted until 2026-09-17 18:14:30.021749558 +0000 UTC m=+2355.549531373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" (UID: "53c07399-ba10-4375-a38e-3d50694d1cd3") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 18:14:29.522003 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.521958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53c07399-ba10-4375-a38e-3d50694d1cd3-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.522270 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.522251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.522306 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.522250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.532228 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.532205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzw2\" (UniqueName: \"kubernetes.io/projected/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-api-access-kvzw2\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:29.602940 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:29.602869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:30.026192 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:30.026156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:30.028710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:30.028680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:30.218947 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:30.218910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:30.493362 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:30.493288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"]
Sep 17 18:14:30.496137 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:14:30.496100    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c07399_ba10_4375_a38e_3d50694d1cd3.slice/crio-ba524c90fab8be192877ce42479d2f24b61fc6826e242d1a3f5aaf050997fb25 WatchSource:0}: Error finding container ba524c90fab8be192877ce42479d2f24b61fc6826e242d1a3f5aaf050997fb25: Status 404 returned error can't find the container with id ba524c90fab8be192877ce42479d2f24b61fc6826e242d1a3f5aaf050997fb25
Sep 17 18:14:31.356212 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:31.356177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerStarted","Data":"4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539"}
Sep 17 18:14:31.356407 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:31.356233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerStarted","Data":"ba524c90fab8be192877ce42479d2f24b61fc6826e242d1a3f5aaf050997fb25"}
Sep 17 18:14:34.371692 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:34.371660    2563 generic.go:356] "Generic (PLEG): container finished" podID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerID="4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539" exitCode=0
Sep 17 18:14:34.372071 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:34.371731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerDied","Data":"4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539"}
Sep 17 18:14:35.378850 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:35.378813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerStarted","Data":"a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9"}
Sep 17 18:14:35.379270 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:35.378878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerStarted","Data":"b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f"}
Sep 17 18:14:35.379270 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:35.379066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:35.404238 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:35.404184    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" podStartSLOduration=2.664925577 podStartE2EDuration="6.404163003s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739237426s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:35.402975178 +0000 UTC m=+2360.930757004" watchObservedRunningTime="2026-09-17 18:14:35.404163003 +0000 UTC m=+2360.931944831"
Sep 17 18:14:36.383838 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:36.383804    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:37.160399 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:37.160359    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:14:42.393269 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:42.393237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:14:47.159756 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:47.159711    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:14:57.159720 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:14:57.159676    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:15:07.161000 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:07.160965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:15:12.397812 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:12.397769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:15:14.430181 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:14.430145    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"]
Sep 17 18:15:14.430572 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:14.430419    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" containerID="cri-o://3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591" gracePeriod=30
Sep 17 18:15:14.430572 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:14.430490    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kube-rbac-proxy" containerID="cri-o://f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e" gracePeriod=30
Sep 17 18:15:14.452994 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:14.452964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:14.574957 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:14.574925    2563 generic.go:356] "Generic (PLEG): container finished" podID="90403477-a318-4382-b20c-b5c5c70c102e" containerID="f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e" exitCode=2
Sep 17 18:15:14.575101 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:14.575004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerDied","Data":"f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e"}
Sep 17 18:15:17.154124 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:17.154079    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 18:15:17.160152 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:17.160125    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 18:15:17.766024 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:17.765988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:19.178654 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.178627    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:15:19.269670 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.269597    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-kube-rbac-proxy-sar-config\") pod \"90403477-a318-4382-b20c-b5c5c70c102e\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") "
Sep 17 18:15:19.269670 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.269655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-cabundle-cert\") pod \"90403477-a318-4382-b20c-b5c5c70c102e\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") "
Sep 17 18:15:19.269894 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.269688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg4hn\" (UniqueName: \"kubernetes.io/projected/90403477-a318-4382-b20c-b5c5c70c102e-kube-api-access-mg4hn\") pod \"90403477-a318-4382-b20c-b5c5c70c102e\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") "
Sep 17 18:15:19.269894 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.269733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90403477-a318-4382-b20c-b5c5c70c102e-proxy-tls\") pod \"90403477-a318-4382-b20c-b5c5c70c102e\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") "
Sep 17 18:15:19.269894 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.269763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90403477-a318-4382-b20c-b5c5c70c102e-kserve-provision-location\") pod \"90403477-a318-4382-b20c-b5c5c70c102e\" (UID: \"90403477-a318-4382-b20c-b5c5c70c102e\") "
Sep 17 18:15:19.270079 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.270055    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90403477-a318-4382-b20c-b5c5c70c102e" (UID: "90403477-a318-4382-b20c-b5c5c70c102e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:19.270123 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.270034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "90403477-a318-4382-b20c-b5c5c70c102e" (UID: "90403477-a318-4382-b20c-b5c5c70c102e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:19.270167 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.270146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90403477-a318-4382-b20c-b5c5c70c102e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90403477-a318-4382-b20c-b5c5c70c102e" (UID: "90403477-a318-4382-b20c-b5c5c70c102e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:19.271949 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.271917    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90403477-a318-4382-b20c-b5c5c70c102e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90403477-a318-4382-b20c-b5c5c70c102e" (UID: "90403477-a318-4382-b20c-b5c5c70c102e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:19.272052 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.271975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90403477-a318-4382-b20c-b5c5c70c102e-kube-api-access-mg4hn" (OuterVolumeSpecName: "kube-api-access-mg4hn") pod "90403477-a318-4382-b20c-b5c5c70c102e" (UID: "90403477-a318-4382-b20c-b5c5c70c102e"). InnerVolumeSpecName "kube-api-access-mg4hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:19.356203 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.356162    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"]
Sep 17 18:15:19.356500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.356468    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kserve-container" containerID="cri-o://b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f" gracePeriod=30
Sep 17 18:15:19.356664 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.356504    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kube-rbac-proxy" containerID="cri-o://a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9" gracePeriod=30
Sep 17 18:15:19.370391 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.370368    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4hn\" (UniqueName: \"kubernetes.io/projected/90403477-a318-4382-b20c-b5c5c70c102e-kube-api-access-mg4hn\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.370510 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.370393    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90403477-a318-4382-b20c-b5c5c70c102e-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.370510 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.370410    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90403477-a318-4382-b20c-b5c5c70c102e-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.370510 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.370422    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.370510 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.370436    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90403477-a318-4382-b20c-b5c5c70c102e-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.376868 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.376834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:19.571089 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.571001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:19.601596 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.601560    2563 generic.go:356] "Generic (PLEG): container finished" podID="90403477-a318-4382-b20c-b5c5c70c102e" containerID="3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591" exitCode=0
Sep 17 18:15:19.601774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.601592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerDied","Data":"3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591"}
Sep 17 18:15:19.601774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.601627    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"
Sep 17 18:15:19.601774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.601661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn" event={"ID":"90403477-a318-4382-b20c-b5c5c70c102e","Type":"ContainerDied","Data":"8f3e9f9184944f8b5f74b91d737c3982d205e2525967a42ef25dadd84b8ef26c"}
Sep 17 18:15:19.601774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.601689    2563 scope.go:117] "RemoveContainer" containerID="f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e"
Sep 17 18:15:19.605298 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.605269    2563 generic.go:356] "Generic (PLEG): container finished" podID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerID="a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9" exitCode=2
Sep 17 18:15:19.605421 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.605352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerDied","Data":"a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9"}
Sep 17 18:15:19.612115 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.612096    2563 scope.go:117] "RemoveContainer" containerID="3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591"
Sep 17 18:15:19.621489 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.621470    2563 scope.go:117] "RemoveContainer" containerID="a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683"
Sep 17 18:15:19.629286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.629265    2563 scope.go:117] "RemoveContainer" containerID="f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e"
Sep 17 18:15:19.629565 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:15:19.629546    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e\": container with ID starting with f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e not found: ID does not exist" containerID="f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e"
Sep 17 18:15:19.629648 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.629578    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e"} err="failed to get container status \"f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e\": rpc error: code = NotFound desc = could not find container \"f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e\": container with ID starting with f2cbbd2d41922bd5664643c9f3d3752ce204931c97f497d56c5ffbaedf6ae49e not found: ID does not exist"
Sep 17 18:15:19.629648 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.629603    2563 scope.go:117] "RemoveContainer" containerID="3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591"
Sep 17 18:15:19.629900 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:15:19.629883    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591\": container with ID starting with 3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591 not found: ID does not exist" containerID="3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591"
Sep 17 18:15:19.629982 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.629909    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591"} err="failed to get container status \"3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591\": rpc error: code = NotFound desc = could not find container \"3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591\": container with ID starting with 3e723bec6f41401fc4aaa0a06ea47ccfb10899bf0b536d9ec38b02e50fd4f591 not found: ID does not exist"
Sep 17 18:15:19.629982 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.629933    2563 scope.go:117] "RemoveContainer" containerID="a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683"
Sep 17 18:15:19.630196 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:15:19.630179    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683\": container with ID starting with a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683 not found: ID does not exist" containerID="a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683"
Sep 17 18:15:19.630247 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.630203    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683"} err="failed to get container status \"a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683\": rpc error: code = NotFound desc = could not find container \"a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683\": container with ID starting with a3fb679e21c07902071183c1c6cca63943c7773485ea62dd5708682a98f48683 not found: ID does not exist"
Sep 17 18:15:19.642819 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.642586    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"]
Sep 17 18:15:19.644244 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.644206    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-wgwrn"]
Sep 17 18:15:19.646375 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.646299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:19.695729 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.695686    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"]
Sep 17 18:15:19.696180 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696160    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container"
Sep 17 18:15:19.696276 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696182    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container"
Sep 17 18:15:19.696276 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696199    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kube-rbac-proxy"
Sep 17 18:15:19.696276 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696207    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kube-rbac-proxy"
Sep 17 18:15:19.696276 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696247    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="storage-initializer"
Sep 17 18:15:19.696276 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="storage-initializer"
Sep 17 18:15:19.696541 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696352    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kube-rbac-proxy"
Sep 17 18:15:19.696541 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.696365    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90403477-a318-4382-b20c-b5c5c70c102e" containerName="kserve-container"
Sep 17 18:15:19.701158 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.701135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.703565 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.703540    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:15:19.703728 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.703588    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:15:19.708590 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.708569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"]
Sep 17 18:15:19.775496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.775464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.775688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.775513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e1a8fa-bbdb-4717-9084-08ec3593a350-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.775688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.775565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.775688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.775601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.775688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.775664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2c5c\" (UniqueName: \"kubernetes.io/projected/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-api-access-j2c5c\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.813717 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.813687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:19.876186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.876186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e1a8fa-bbdb-4717-9084-08ec3593a350-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.876186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.876458 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.876458 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2c5c\" (UniqueName: \"kubernetes.io/projected/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-api-access-j2c5c\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.876724 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.876949 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.876923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.877064 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.877043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.879033 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.879015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e1a8fa-bbdb-4717-9084-08ec3593a350-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:19.885112 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:19.885081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2c5c\" (UniqueName: \"kubernetes.io/projected/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-api-access-j2c5c\") pod \"isvc-canary-stop-predictor-684df5466b-fvvc9\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:20.013250 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:20.013217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:20.278294 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:20.278264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:20.294592 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:20.294544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"]
Sep 17 18:15:20.296873 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:15:20.296841    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e1a8fa_bbdb_4717_9084_08ec3593a350.slice/crio-c555da4b0275dddd75df5e6fbb71d94ca160ef07505a77601620faf914bbdc5b WatchSource:0}: Error finding container c555da4b0275dddd75df5e6fbb71d94ca160ef07505a77601620faf914bbdc5b: Status 404 returned error can't find the container with id c555da4b0275dddd75df5e6fbb71d94ca160ef07505a77601620faf914bbdc5b
Sep 17 18:15:20.613739 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:20.613645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerStarted","Data":"f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3"}
Sep 17 18:15:20.613739 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:20.613728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerStarted","Data":"c555da4b0275dddd75df5e6fbb71d94ca160ef07505a77601620faf914bbdc5b"}
Sep 17 18:15:20.957917 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:20.957883    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90403477-a318-4382-b20c-b5c5c70c102e" path="/var/lib/kubelet/pods/90403477-a318-4382-b20c-b5c5c70c102e/volumes"
Sep 17 18:15:22.389268 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:22.389225    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 17 18:15:22.394647 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:22.394624    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.51:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:15:24.495443 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.495398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:24.522456 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.522421    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"]
Sep 17 18:15:24.526206 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.526185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.528470 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.528441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 18:15:24.528470 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.528455    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 18:15:24.536703 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.536681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"]
Sep 17 18:15:24.623717 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.623674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab85df70-e129-42f0-b125-f4fc2046bda2-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.623931 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.623731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab85df70-e129-42f0-b125-f4fc2046bda2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.623931 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.623821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdfx\" (UniqueName: \"kubernetes.io/projected/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-api-access-lsdfx\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.623931 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.623856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.623931 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.623884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.633695 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.633669    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerID="f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3" exitCode=0
Sep 17 18:15:24.633848 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.633703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerDied","Data":"f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3"}
Sep 17 18:15:24.724925 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.724889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab85df70-e129-42f0-b125-f4fc2046bda2-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.725050 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.724980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab85df70-e129-42f0-b125-f4fc2046bda2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.725115 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.725069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdfx\" (UniqueName: \"kubernetes.io/projected/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-api-access-lsdfx\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.725115 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.725102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.725230 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.725145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.726048 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.725951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab85df70-e129-42f0-b125-f4fc2046bda2-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.726305 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.726278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.726552 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.726491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.728240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.728222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab85df70-e129-42f0-b125-f4fc2046bda2-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.733274 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.733253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdfx\" (UniqueName: \"kubernetes.io/projected/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-api-access-lsdfx\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:24.742031 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.742001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:24.837963 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:24.837932    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:25.114290 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.114124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"]
Sep 17 18:15:25.116976 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:15:25.116944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab85df70_e129_42f0_b125_f4fc2046bda2.slice/crio-b6c4434aca5208f273dc854d70f39fe8f156f7af87e23697c74e1d448b56ab59 WatchSource:0}: Error finding container b6c4434aca5208f273dc854d70f39fe8f156f7af87e23697c74e1d448b56ab59: Status 404 returned error can't find the container with id b6c4434aca5208f273dc854d70f39fe8f156f7af87e23697c74e1d448b56ab59
Sep 17 18:15:25.640870 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.640827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerStarted","Data":"c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61"}
Sep 17 18:15:25.641357 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.640904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerStarted","Data":"b6c4434aca5208f273dc854d70f39fe8f156f7af87e23697c74e1d448b56ab59"}
Sep 17 18:15:25.643643 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.643613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerStarted","Data":"36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f"}
Sep 17 18:15:25.643785 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.643676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerStarted","Data":"20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081"}
Sep 17 18:15:25.643890 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.643866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:25.643952 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.643908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:25.645775 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.645749    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:15:25.678661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:25.678607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podStartSLOduration=3.088868148 podStartE2EDuration="6.678588032s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.589719884s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:25.678019545 +0000 UTC m=+2411.205801370" watchObservedRunningTime="2026-09-17 18:15:25.678588032 +0000 UTC m=+2411.206369858"
Sep 17 18:15:26.413012 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.412987    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:15:26.437663 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.437638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-rbac-proxy-sar-config\") pod \"53c07399-ba10-4375-a38e-3d50694d1cd3\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") "
Sep 17 18:15:26.437780 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.437734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls\") pod \"53c07399-ba10-4375-a38e-3d50694d1cd3\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") "
Sep 17 18:15:26.437780 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.437762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-cabundle-cert\") pod \"53c07399-ba10-4375-a38e-3d50694d1cd3\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") "
Sep 17 18:15:26.437904 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.437822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53c07399-ba10-4375-a38e-3d50694d1cd3-kserve-provision-location\") pod \"53c07399-ba10-4375-a38e-3d50694d1cd3\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") "
Sep 17 18:15:26.437904 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.437878    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvzw2\" (UniqueName: \"kubernetes.io/projected/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-api-access-kvzw2\") pod \"53c07399-ba10-4375-a38e-3d50694d1cd3\" (UID: \"53c07399-ba10-4375-a38e-3d50694d1cd3\") "
Sep 17 18:15:26.438109 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.438074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53c07399-ba10-4375-a38e-3d50694d1cd3" (UID: "53c07399-ba10-4375-a38e-3d50694d1cd3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:26.438237 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.438175    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:26.438391 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.438362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "53c07399-ba10-4375-a38e-3d50694d1cd3" (UID: "53c07399-ba10-4375-a38e-3d50694d1cd3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:26.438494 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.438423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53c07399-ba10-4375-a38e-3d50694d1cd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "53c07399-ba10-4375-a38e-3d50694d1cd3" (UID: "53c07399-ba10-4375-a38e-3d50694d1cd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:26.439957 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.439933    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53c07399-ba10-4375-a38e-3d50694d1cd3" (UID: "53c07399-ba10-4375-a38e-3d50694d1cd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:26.440527 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.440510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-api-access-kvzw2" (OuterVolumeSpecName: "kube-api-access-kvzw2") pod "53c07399-ba10-4375-a38e-3d50694d1cd3" (UID: "53c07399-ba10-4375-a38e-3d50694d1cd3"). InnerVolumeSpecName "kube-api-access-kvzw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:26.538923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.538844    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53c07399-ba10-4375-a38e-3d50694d1cd3-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:26.538923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.538873    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/53c07399-ba10-4375-a38e-3d50694d1cd3-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:26.538923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.538883    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/53c07399-ba10-4375-a38e-3d50694d1cd3-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:26.538923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.538892    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvzw2\" (UniqueName: \"kubernetes.io/projected/53c07399-ba10-4375-a38e-3d50694d1cd3-kube-api-access-kvzw2\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:15:26.650508 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.650472    2563 generic.go:356] "Generic (PLEG): container finished" podID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerID="b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f" exitCode=0
Sep 17 18:15:26.650913 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.650540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"
Sep 17 18:15:26.650913 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.650543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerDied","Data":"b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f"}
Sep 17 18:15:26.650913 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.650662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs" event={"ID":"53c07399-ba10-4375-a38e-3d50694d1cd3","Type":"ContainerDied","Data":"ba524c90fab8be192877ce42479d2f24b61fc6826e242d1a3f5aaf050997fb25"}
Sep 17 18:15:26.650913 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.650694    2563 scope.go:117] "RemoveContainer" containerID="a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9"
Sep 17 18:15:26.651272 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.651236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:15:26.660207 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.660192    2563 scope.go:117] "RemoveContainer" containerID="b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f"
Sep 17 18:15:26.668410 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.668393    2563 scope.go:117] "RemoveContainer" containerID="4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539"
Sep 17 18:15:26.672212 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.672190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"]
Sep 17 18:15:26.676143 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.676118    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-rhpbs"]
Sep 17 18:15:26.676650 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.676636    2563 scope.go:117] "RemoveContainer" containerID="a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9"
Sep 17 18:15:26.676938 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:15:26.676915    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9\": container with ID starting with a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9 not found: ID does not exist" containerID="a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9"
Sep 17 18:15:26.677008 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.676945    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9"} err="failed to get container status \"a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9\": rpc error: code = NotFound desc = could not find container \"a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9\": container with ID starting with a8545186b45bd7551e15d14caa6d8ebf421aa4459e1c1865e6c5933ca1fba0e9 not found: ID does not exist"
Sep 17 18:15:26.677008 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.676961    2563 scope.go:117] "RemoveContainer" containerID="b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f"
Sep 17 18:15:26.677189 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:15:26.677173    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f\": container with ID starting with b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f not found: ID does not exist" containerID="b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f"
Sep 17 18:15:26.677230 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.677194    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f"} err="failed to get container status \"b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f\": rpc error: code = NotFound desc = could not find container \"b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f\": container with ID starting with b66c28e7b2bbc7b140ef8902581d0be3f08ba339ae6ad163c63e4184963e927f not found: ID does not exist"
Sep 17 18:15:26.677230 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.677209    2563 scope.go:117] "RemoveContainer" containerID="4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539"
Sep 17 18:15:26.677427 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:15:26.677412    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539\": container with ID starting with 4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539 not found: ID does not exist" containerID="4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539"
Sep 17 18:15:26.677469 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.677431    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539"} err="failed to get container status \"4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539\": rpc error: code = NotFound desc = could not find container \"4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539\": container with ID starting with 4d7d0b5dc2a6442169068c895bef98eda1abb86100db611fe1959ba39cf5b539 not found: ID does not exist"
Sep 17 18:15:26.958815 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:26.958768    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" path="/var/lib/kubelet/pods/53c07399-ba10-4375-a38e-3d50694d1cd3/volumes"
Sep 17 18:15:29.668047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:29.667961    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerID="c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61" exitCode=0
Sep 17 18:15:29.668047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:29.668002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerDied","Data":"c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61"}
Sep 17 18:15:30.675523 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:30.675485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerStarted","Data":"236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292"}
Sep 17 18:15:30.675912 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:30.675582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerStarted","Data":"964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62"}
Sep 17 18:15:30.675912 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:30.675810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:30.698249 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:30.698199    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" podStartSLOduration=2.60543647 podStartE2EDuration="6.698182294s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.092745824s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:30.697320633 +0000 UTC m=+2416.225102459" watchObservedRunningTime="2026-09-17 18:15:30.698182294 +0000 UTC m=+2416.225964121"
Sep 17 18:15:31.656054 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:31.656021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:15:31.656579 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:31.656552    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:15:31.683096 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:31.683070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:33.330154 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:33.330044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:15:33.355690 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:33.335918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:15:37.692992 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:37.692908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:15:41.656781 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:41.656737    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:15:44.571718 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:44.571676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:51.657143 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:15:51.657097    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:16:01.656561 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:01.656511    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:16:07.697996 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:07.697965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:16:11.657486 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:11.657442    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:16:14.569486 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:14.569444    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"]
Sep 17 18:16:14.569914 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:14.569708    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kserve-container" containerID="cri-o://964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62" gracePeriod=30
Sep 17 18:16:14.569914 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:14.569743    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kube-rbac-proxy" containerID="cri-o://236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292" gracePeriod=30
Sep 17 18:16:14.593002 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:14.592968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:14.895354 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:14.895274    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerID="236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292" exitCode=2
Sep 17 18:16:14.895354 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:14.895346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerDied","Data":"236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292"}
Sep 17 18:16:17.688617 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:17.688566    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 17 18:16:17.694319 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:17.694290    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:16:19.740626 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:19.740591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.818192 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:19.818156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:21.320862 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.320839    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:16:21.421405 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421309    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsdfx\" (UniqueName: \"kubernetes.io/projected/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-api-access-lsdfx\") pod \"ab85df70-e129-42f0-b125-f4fc2046bda2\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") "
Sep 17 18:16:21.421405 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421355    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab85df70-e129-42f0-b125-f4fc2046bda2-kserve-provision-location\") pod \"ab85df70-e129-42f0-b125-f4fc2046bda2\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") "
Sep 17 18:16:21.421405 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-cabundle-cert\") pod \"ab85df70-e129-42f0-b125-f4fc2046bda2\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") "
Sep 17 18:16:21.421688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab85df70-e129-42f0-b125-f4fc2046bda2-proxy-tls\") pod \"ab85df70-e129-42f0-b125-f4fc2046bda2\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") "
Sep 17 18:16:21.421688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-rbac-proxy-sar-config\") pod \"ab85df70-e129-42f0-b125-f4fc2046bda2\" (UID: \"ab85df70-e129-42f0-b125-f4fc2046bda2\") "
Sep 17 18:16:21.421861 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421749    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab85df70-e129-42f0-b125-f4fc2046bda2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab85df70-e129-42f0-b125-f4fc2046bda2" (UID: "ab85df70-e129-42f0-b125-f4fc2046bda2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:21.421918 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab85df70-e129-42f0-b125-f4fc2046bda2" (UID: "ab85df70-e129-42f0-b125-f4fc2046bda2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:21.422000 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.421978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab85df70-e129-42f0-b125-f4fc2046bda2" (UID: "ab85df70-e129-42f0-b125-f4fc2046bda2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:21.423465 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.423439    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-api-access-lsdfx" (OuterVolumeSpecName: "kube-api-access-lsdfx") pod "ab85df70-e129-42f0-b125-f4fc2046bda2" (UID: "ab85df70-e129-42f0-b125-f4fc2046bda2"). InnerVolumeSpecName "kube-api-access-lsdfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:21.423578 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.423521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab85df70-e129-42f0-b125-f4fc2046bda2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab85df70-e129-42f0-b125-f4fc2046bda2" (UID: "ab85df70-e129-42f0-b125-f4fc2046bda2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:21.522190 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.522158    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsdfx\" (UniqueName: \"kubernetes.io/projected/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-api-access-lsdfx\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:21.522190 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.522185    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab85df70-e129-42f0-b125-f4fc2046bda2-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:21.522190 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.522197    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:21.522417 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.522207    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab85df70-e129-42f0-b125-f4fc2046bda2-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:21.522417 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.522216    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab85df70-e129-42f0-b125-f4fc2046bda2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:21.656459 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.656417    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:16:21.931942 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.931909    2563 generic.go:356] "Generic (PLEG): container finished" podID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerID="964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62" exitCode=0
Sep 17 18:16:21.932141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.931981    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"
Sep 17 18:16:21.932141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.931992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerDied","Data":"964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62"}
Sep 17 18:16:21.932141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.932078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j" event={"ID":"ab85df70-e129-42f0-b125-f4fc2046bda2","Type":"ContainerDied","Data":"b6c4434aca5208f273dc854d70f39fe8f156f7af87e23697c74e1d448b56ab59"}
Sep 17 18:16:21.932141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.932104    2563 scope.go:117] "RemoveContainer" containerID="236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292"
Sep 17 18:16:21.941683 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.941651    2563 scope.go:117] "RemoveContainer" containerID="964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62"
Sep 17 18:16:21.949630 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.949608    2563 scope.go:117] "RemoveContainer" containerID="c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61"
Sep 17 18:16:21.957086 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.957069    2563 scope.go:117] "RemoveContainer" containerID="236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292"
Sep 17 18:16:21.957384 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:16:21.957364    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292\": container with ID starting with 236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292 not found: ID does not exist" containerID="236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292"
Sep 17 18:16:21.957441 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.957393    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292"} err="failed to get container status \"236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292\": rpc error: code = NotFound desc = could not find container \"236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292\": container with ID starting with 236a0f4037361b467dc5c3e398d674eb29d0a5d9877beffecbf3a6b95fdda292 not found: ID does not exist"
Sep 17 18:16:21.957441 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.957412    2563 scope.go:117] "RemoveContainer" containerID="964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62"
Sep 17 18:16:21.957678 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:16:21.957663    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62\": container with ID starting with 964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62 not found: ID does not exist" containerID="964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62"
Sep 17 18:16:21.957735 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.957681    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62"} err="failed to get container status \"964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62\": rpc error: code = NotFound desc = could not find container \"964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62\": container with ID starting with 964a0d5370ddbf2cab5c0398677eb3f2f05b146fade706e478054149aa7b7b62 not found: ID does not exist"
Sep 17 18:16:21.957735 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.957695    2563 scope.go:117] "RemoveContainer" containerID="c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61"
Sep 17 18:16:21.957915 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:16:21.957899    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61\": container with ID starting with c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61 not found: ID does not exist" containerID="c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61"
Sep 17 18:16:21.957956 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.957917    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61"} err="failed to get container status \"c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61\": rpc error: code = NotFound desc = could not find container \"c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61\": container with ID starting with c0bd5d0995fb614e8d16fe4eaf0e61b192c3696acd3edf44cecb12520f657a61 not found: ID does not exist"
Sep 17 18:16:21.964544 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.964522    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"]
Sep 17 18:16:21.970355 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:21.970336    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-8jx7j"]
Sep 17 18:16:22.958492 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:22.958457    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" path="/var/lib/kubelet/pods/ab85df70-e129-42f0-b125-f4fc2046bda2/volumes"
Sep 17 18:16:31.656679 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:31.656637    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:16:41.657747 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:41.657710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:16:49.647228 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:49.647192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:49.661207 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:49.661178    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"]
Sep 17 18:16:49.661436 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:49.661412    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" containerID="cri-o://20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081" gracePeriod=30
Sep 17 18:16:49.661550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:49.661505    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kube-rbac-proxy" containerID="cri-o://36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f" gracePeriod=30
Sep 17 18:16:49.748423 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:49.748391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:50.067533 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:50.067500    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerID="36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f" exitCode=2
Sep 17 18:16:50.067710 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:50.067554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerDied","Data":"36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f"}
Sep 17 18:16:51.651408 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:51.651367    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 18:16:51.656800 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:51.656763    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:16:54.512555 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.512529    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:16:54.610832 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.610740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-cabundle-cert\") pod \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") "
Sep 17 18:16:54.610832 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.610809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-rbac-proxy-sar-config\") pod \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") "
Sep 17 18:16:54.611067 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.610840    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e1a8fa-bbdb-4717-9084-08ec3593a350-proxy-tls\") pod \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") "
Sep 17 18:16:54.611067 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.610885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2c5c\" (UniqueName: \"kubernetes.io/projected/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-api-access-j2c5c\") pod \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") "
Sep 17 18:16:54.611067 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.610926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kserve-provision-location\") pod \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\" (UID: \"b4e1a8fa-bbdb-4717-9084-08ec3593a350\") "
Sep 17 18:16:54.611423 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.611192    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4e1a8fa-bbdb-4717-9084-08ec3593a350" (UID: "b4e1a8fa-bbdb-4717-9084-08ec3593a350"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:54.611423 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.611320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4e1a8fa-bbdb-4717-9084-08ec3593a350" (UID: "b4e1a8fa-bbdb-4717-9084-08ec3593a350"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:54.611423 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.611367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4e1a8fa-bbdb-4717-9084-08ec3593a350" (UID: "b4e1a8fa-bbdb-4717-9084-08ec3593a350"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:54.611423 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.611380    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.613184 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.613156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e1a8fa-bbdb-4717-9084-08ec3593a350-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4e1a8fa-bbdb-4717-9084-08ec3593a350" (UID: "b4e1a8fa-bbdb-4717-9084-08ec3593a350"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:54.613311 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.613291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-api-access-j2c5c" (OuterVolumeSpecName: "kube-api-access-j2c5c") pod "b4e1a8fa-bbdb-4717-9084-08ec3593a350" (UID: "b4e1a8fa-bbdb-4717-9084-08ec3593a350"). InnerVolumeSpecName "kube-api-access-j2c5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:54.712145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.712108    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.712145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.712137    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e1a8fa-bbdb-4717-9084-08ec3593a350-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.712145 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.712148    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2c5c\" (UniqueName: \"kubernetes.io/projected/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kube-api-access-j2c5c\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.712368 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:54.712157    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4e1a8fa-bbdb-4717-9084-08ec3593a350-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:16:55.092956 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.092914    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerID="20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081" exitCode=0
Sep 17 18:16:55.093186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.092980    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"
Sep 17 18:16:55.093186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.092993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerDied","Data":"20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081"}
Sep 17 18:16:55.093186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.093057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9" event={"ID":"b4e1a8fa-bbdb-4717-9084-08ec3593a350","Type":"ContainerDied","Data":"c555da4b0275dddd75df5e6fbb71d94ca160ef07505a77601620faf914bbdc5b"}
Sep 17 18:16:55.093186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.093095    2563 scope.go:117] "RemoveContainer" containerID="36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f"
Sep 17 18:16:55.101918 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.101894    2563 scope.go:117] "RemoveContainer" containerID="20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081"
Sep 17 18:16:55.109868 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.109844    2563 scope.go:117] "RemoveContainer" containerID="f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3"
Sep 17 18:16:55.112333 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.112310    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"]
Sep 17 18:16:55.118166 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.118142    2563 scope.go:117] "RemoveContainer" containerID="36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f"
Sep 17 18:16:55.118450 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:16:55.118429    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f\": container with ID starting with 36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f not found: ID does not exist" containerID="36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f"
Sep 17 18:16:55.118528 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.118459    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f"} err="failed to get container status \"36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f\": rpc error: code = NotFound desc = could not find container \"36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f\": container with ID starting with 36bb0211afc9ee183f896d919f65c9ad62e615c4f5c8e7a57b75c3a7fb507c4f not found: ID does not exist"
Sep 17 18:16:55.118528 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.118479    2563 scope.go:117] "RemoveContainer" containerID="20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081"
Sep 17 18:16:55.118751 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:16:55.118732    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081\": container with ID starting with 20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081 not found: ID does not exist" containerID="20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081"
Sep 17 18:16:55.118834 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.118756    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081"} err="failed to get container status \"20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081\": rpc error: code = NotFound desc = could not find container \"20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081\": container with ID starting with 20c20ba68a7e1df5c2e804e7825b8a86ee70db9e5bfc378a7ced94aadd4ae081 not found: ID does not exist"
Sep 17 18:16:55.118834 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.118772    2563 scope.go:117] "RemoveContainer" containerID="f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3"
Sep 17 18:16:55.119069 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:16:55.119049    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3\": container with ID starting with f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3 not found: ID does not exist" containerID="f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3"
Sep 17 18:16:55.119129 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.119074    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3"} err="failed to get container status \"f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3\": rpc error: code = NotFound desc = could not find container \"f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3\": container with ID starting with f0802d47d0483fa974d78bf6e920758f6d80c649b6f4f9a705860e6f0088c7c3 not found: ID does not exist"
Sep 17 18:16:55.119220 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:55.119200    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-fvvc9"]
Sep 17 18:16:56.958999 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:56.958963    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" path="/var/lib/kubelet/pods/b4e1a8fa-bbdb-4717-9084-08ec3593a350/volumes"
Sep 17 18:16:56.963542 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:56.963513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:59.888725 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.888689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:59.927574 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.927537    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"]
Sep 17 18:16:59.928135 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928135 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928134    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928156    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="storage-initializer"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928165    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="storage-initializer"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928181    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kserve-container"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928190    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kserve-container"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928203    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="storage-initializer"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928211    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="storage-initializer"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928220    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kserve-container"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928228    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kserve-container"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928249    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928269    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="storage-initializer"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928277    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="storage-initializer"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928286    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928294    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928307 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kserve-container"
Sep 17 18:16:59.928810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928402    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928413    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c07399-ba10-4375-a38e-3d50694d1cd3" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928424    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e1a8fa-bbdb-4717-9084-08ec3593a350" containerName="kserve-container"
Sep 17 18:16:59.928810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928433    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kube-rbac-proxy"
Sep 17 18:16:59.928810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.928440    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab85df70-e129-42f0-b125-f4fc2046bda2" containerName="kserve-container"
Sep 17 18:16:59.933501 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.933482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:16:59.935758 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.935731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 18:16:59.936020 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.935999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:16:59.936137 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.936012    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 18:16:59.936226 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.935999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:16:59.936226 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.936005    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:16:59.936348 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.936008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:16:59.936348 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.936000    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:16:59.939087 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.939063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"]
Sep 17 18:16:59.950956 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.950930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:16:59.951095 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.950963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkk5z\" (UniqueName: \"kubernetes.io/projected/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-api-access-xkk5z\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:16:59.951095 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.950998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:16:59.951095 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.951021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:16:59.951095 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:16:59.951074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f63ac8a-5b96-47b1-a975-5b0c92d34619-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.052075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.052075 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkk5z\" (UniqueName: \"kubernetes.io/projected/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-api-access-xkk5z\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.052364 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.052364 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.052364 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f63ac8a-5b96-47b1-a975-5b0c92d34619-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.052539 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.053011 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.052984    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.053146 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.053107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.055155 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.055125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f63ac8a-5b96-47b1-a975-5b0c92d34619-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.063420 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.063391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkk5z\" (UniqueName: \"kubernetes.io/projected/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-api-access-xkk5z\") pod \"isvc-lightgbm-predictor-54f85ddf88-6v488\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.110844 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.110778    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:00.245081 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.245044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:00.521189 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:00.521162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"]
Sep 17 18:17:00.522777 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:17:00.522747    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f63ac8a_5b96_47b1_a975_5b0c92d34619.slice/crio-9fef271b77f94528081fc3037530a84b561bbd6479c4bcd7d0e30b8a4c107ec9 WatchSource:0}: Error finding container 9fef271b77f94528081fc3037530a84b561bbd6479c4bcd7d0e30b8a4c107ec9: Status 404 returned error can't find the container with id 9fef271b77f94528081fc3037530a84b561bbd6479c4bcd7d0e30b8a4c107ec9
Sep 17 18:17:01.124931 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:01.124895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerStarted","Data":"cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6"}
Sep 17 18:17:01.125370 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:01.124950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerStarted","Data":"9fef271b77f94528081fc3037530a84b561bbd6479c4bcd7d0e30b8a4c107ec9"}
Sep 17 18:17:05.144874 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:05.144838    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerID="cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6" exitCode=0
Sep 17 18:17:05.145259 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:05.144912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerDied","Data":"cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6"}
Sep 17 18:17:05.146248 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:05.146227    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:17:08.198305 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:08.198209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:13.191293 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:13.191255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerStarted","Data":"94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445"}
Sep 17 18:17:13.191684 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:13.191323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerStarted","Data":"32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210"}
Sep 17 18:17:13.191684 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:13.191520    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:13.211541 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:13.211493    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podStartSLOduration=3.095333346 podStartE2EDuration="14.211478954s" totalImagesPullingTime="7.187493663s" totalInitContainerRuntime="3.928651945s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:13.209116753 +0000 UTC m=+2518.736898583" watchObservedRunningTime="2026-09-17 18:17:13.211478954 +0000 UTC m=+2518.739260779"
Sep 17 18:17:14.196054 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:14.196021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:14.197496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:14.197460    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:17:15.200003 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:15.199762    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:17:20.210167 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:20.210138    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:17:20.210645 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:20.210615    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:17:30.210606 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:30.210566    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:17:40.211496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:40.211424    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:17:40.317596 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:40.317547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.455606 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.455561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.493668 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.493633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"]
Sep 17 18:17:45.602159 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.602121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.607774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.607752    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"]
Sep 17 18:17:45.607914 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.607899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.610377 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.610359    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:17:45.610452 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.610357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 18:17:45.667892 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.667861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.668040 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.667919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c489fc4-a269-45ef-bf18-ece988a0b6cc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.668040 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.668019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.668151 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.668053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.668151 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.668083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cg7j\" (UniqueName: \"kubernetes.io/projected/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-api-access-4cg7j\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.769492 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.769405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.769492 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.769449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cg7j\" (UniqueName: \"kubernetes.io/projected/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-api-access-4cg7j\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.769492 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.769471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.769781 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.769507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c489fc4-a269-45ef-bf18-ece988a0b6cc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.769781 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.769554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.769999 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.769972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.770113 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.770096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.770206 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.770188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.772044 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.772026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c489fc4-a269-45ef-bf18-ece988a0b6cc-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.778061 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.778035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cg7j\" (UniqueName: \"kubernetes.io/projected/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-api-access-4cg7j\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:45.918274 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:45.918237    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:46.194403 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:46.194367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"]
Sep 17 18:17:46.196357 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:17:46.196315    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c489fc4_a269_45ef_bf18_ece988a0b6cc.slice/crio-25a74d2d4e27012f8e6c44ae164be312638650d238929e7107d2c2f6e876b20a WatchSource:0}: Error finding container 25a74d2d4e27012f8e6c44ae164be312638650d238929e7107d2c2f6e876b20a: Status 404 returned error can't find the container with id 25a74d2d4e27012f8e6c44ae164be312638650d238929e7107d2c2f6e876b20a
Sep 17 18:17:46.350178 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:46.350138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerStarted","Data":"314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb"}
Sep 17 18:17:46.350380 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:46.350210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerStarted","Data":"25a74d2d4e27012f8e6c44ae164be312638650d238929e7107d2c2f6e876b20a"}
Sep 17 18:17:50.211071 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:50.211026    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:17:50.370455 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:50.370422    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerID="314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb" exitCode=0
Sep 17 18:17:50.370623 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:50.370496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerDied","Data":"314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb"}
Sep 17 18:17:51.377315 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:51.377279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerStarted","Data":"ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523"}
Sep 17 18:17:51.377966 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:51.377366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerStarted","Data":"395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d"}
Sep 17 18:17:51.377966 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:51.377622    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:51.400198 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:51.400148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" podStartSLOduration=2.441405368 podStartE2EDuration="6.400133491s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.958728123s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:51.398547401 +0000 UTC m=+2556.926329238" watchObservedRunningTime="2026-09-17 18:17:51.400133491 +0000 UTC m=+2556.927915316"
Sep 17 18:17:52.382487 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:52.382455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:17:58.392817 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:17:58.392769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:18:00.210766 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:00.210725    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:18:10.210918 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:10.210874    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:18:20.210818 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:20.210752    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:18:28.436399 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:28.436353    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 18:18:30.211344 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:30.211300    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:18:38.396284 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:38.396207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:18:40.211056 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:40.211013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:18:45.550632 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:45.550590    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"]
Sep 17 18:18:45.551124 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:45.550858    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kserve-container" containerID="cri-o://395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d" gracePeriod=30
Sep 17 18:18:45.551124 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:45.550910    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kube-rbac-proxy" containerID="cri-o://ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523" gracePeriod=30
Sep 17 18:18:45.578803 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:45.578742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:46.642192 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:46.642149    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerID="ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523" exitCode=2
Sep 17 18:18:46.642677 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:46.642225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerDied","Data":"ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523"}
Sep 17 18:18:47.954573 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:47.954543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:18:48.388570 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:48.388471    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 17 18:18:50.708617 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:50.708579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:50.971713 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:50.971633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:53.100889 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.100863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:18:53.184810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.184760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cg7j\" (UniqueName: \"kubernetes.io/projected/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-api-access-4cg7j\") pod \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") "
Sep 17 18:18:53.184954 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.184842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-rbac-proxy-sar-config\") pod \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") "
Sep 17 18:18:53.184954 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.184873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c489fc4-a269-45ef-bf18-ece988a0b6cc-proxy-tls\") pod \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") "
Sep 17 18:18:53.184954 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.184918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kserve-provision-location\") pod \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") "
Sep 17 18:18:53.185058 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.184964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-cabundle-cert\") pod \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\" (UID: \"5c489fc4-a269-45ef-bf18-ece988a0b6cc\") "
Sep 17 18:18:53.185266 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.185241    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c489fc4-a269-45ef-bf18-ece988a0b6cc" (UID: "5c489fc4-a269-45ef-bf18-ece988a0b6cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:53.185379 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.185219    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c489fc4-a269-45ef-bf18-ece988a0b6cc" (UID: "5c489fc4-a269-45ef-bf18-ece988a0b6cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:53.185379 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.185353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c489fc4-a269-45ef-bf18-ece988a0b6cc" (UID: "5c489fc4-a269-45ef-bf18-ece988a0b6cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:53.186965 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.186946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-api-access-4cg7j" (OuterVolumeSpecName: "kube-api-access-4cg7j") pod "5c489fc4-a269-45ef-bf18-ece988a0b6cc" (UID: "5c489fc4-a269-45ef-bf18-ece988a0b6cc"). InnerVolumeSpecName "kube-api-access-4cg7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:53.187030 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.186987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c489fc4-a269-45ef-bf18-ece988a0b6cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c489fc4-a269-45ef-bf18-ece988a0b6cc" (UID: "5c489fc4-a269-45ef-bf18-ece988a0b6cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:53.286675 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.286596    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cg7j\" (UniqueName: \"kubernetes.io/projected/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-api-access-4cg7j\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.286675 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.286624    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.286675 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.286639    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c489fc4-a269-45ef-bf18-ece988a0b6cc-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.286675 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.286653    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c489fc4-a269-45ef-bf18-ece988a0b6cc-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.286675 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.286666    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c489fc4-a269-45ef-bf18-ece988a0b6cc-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:18:53.677621 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.677584    2563 generic.go:356] "Generic (PLEG): container finished" podID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerID="395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d" exitCode=0
Sep 17 18:18:53.677848 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.677662    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"
Sep 17 18:18:53.677848 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.677677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerDied","Data":"395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d"}
Sep 17 18:18:53.677848 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.677769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6" event={"ID":"5c489fc4-a269-45ef-bf18-ece988a0b6cc","Type":"ContainerDied","Data":"25a74d2d4e27012f8e6c44ae164be312638650d238929e7107d2c2f6e876b20a"}
Sep 17 18:18:53.677848 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.677817    2563 scope.go:117] "RemoveContainer" containerID="ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523"
Sep 17 18:18:53.686843 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.686824    2563 scope.go:117] "RemoveContainer" containerID="395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d"
Sep 17 18:18:53.694484 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.694463    2563 scope.go:117] "RemoveContainer" containerID="314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb"
Sep 17 18:18:53.699769 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.699749    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"]
Sep 17 18:18:53.702680 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.702662    2563 scope.go:117] "RemoveContainer" containerID="ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523"
Sep 17 18:18:53.703072 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:18:53.703050    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523\": container with ID starting with ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523 not found: ID does not exist" containerID="ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523"
Sep 17 18:18:53.703141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.703081    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523"} err="failed to get container status \"ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523\": rpc error: code = NotFound desc = could not find container \"ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523\": container with ID starting with ec181247180f3cc7a45591b8a8a408825a87a53a892ee328c09b932666cd5523 not found: ID does not exist"
Sep 17 18:18:53.703141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.703101    2563 scope.go:117] "RemoveContainer" containerID="395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d"
Sep 17 18:18:53.703433 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:18:53.703410    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d\": container with ID starting with 395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d not found: ID does not exist" containerID="395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d"
Sep 17 18:18:53.703494 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.703443    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d"} err="failed to get container status \"395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d\": rpc error: code = NotFound desc = could not find container \"395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d\": container with ID starting with 395ba6002dfa263c92150c166677a5e40a02e1402a50e3cd3c402077166a999d not found: ID does not exist"
Sep 17 18:18:53.703494 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.703465    2563 scope.go:117] "RemoveContainer" containerID="314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb"
Sep 17 18:18:53.703689 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.703674    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-skzz6"]
Sep 17 18:18:53.703751 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:18:53.703734    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb\": container with ID starting with 314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb not found: ID does not exist" containerID="314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb"
Sep 17 18:18:53.703800 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:53.703756    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb"} err="failed to get container status \"314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb\": rpc error: code = NotFound desc = could not find container \"314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb\": container with ID starting with 314617e1c5863e02041f850f491b90d34ed72f643f866268299c8c729203bffb not found: ID does not exist"
Sep 17 18:18:54.649610 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:54.649579    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"]
Sep 17 18:18:54.650040 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:54.649820    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" containerID="cri-o://32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210" gracePeriod=30
Sep 17 18:18:54.650040 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:54.649894    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kube-rbac-proxy" containerID="cri-o://94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445" gracePeriod=30
Sep 17 18:18:54.671099 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:54.671073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:54.961230 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:54.961187    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" path="/var/lib/kubelet/pods/5c489fc4-a269-45ef-bf18-ece988a0b6cc/volumes"
Sep 17 18:18:55.200130 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:55.200089    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 17 18:18:55.693159 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:55.693128    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerID="94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445" exitCode=2
Sep 17 18:18:55.693536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:55.693210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerDied","Data":"94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445"}
Sep 17 18:18:57.953545 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:57.953508    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Sep 17 18:18:59.956588 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:18:59.956553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.200886 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.200848    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Sep 17 18:19:00.205142 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.205113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.506041 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.506014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:19:00.651290 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651252    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f63ac8a-5b96-47b1-a975-5b0c92d34619-proxy-tls\") pod \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") "
Sep 17 18:19:00.651500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kserve-provision-location\") pod \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") "
Sep 17 18:19:00.651500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651334    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-cabundle-cert\") pod \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") "
Sep 17 18:19:00.651500 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-rbac-proxy-sar-config\") pod \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") "
Sep 17 18:19:00.651677 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651538    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkk5z\" (UniqueName: \"kubernetes.io/projected/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-api-access-xkk5z\") pod \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\" (UID: \"5f63ac8a-5b96-47b1-a975-5b0c92d34619\") "
Sep 17 18:19:00.651774 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f63ac8a-5b96-47b1-a975-5b0c92d34619" (UID: "5f63ac8a-5b96-47b1-a975-5b0c92d34619"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:00.651849 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651755    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f63ac8a-5b96-47b1-a975-5b0c92d34619" (UID: "5f63ac8a-5b96-47b1-a975-5b0c92d34619"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:00.651892 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f63ac8a-5b96-47b1-a975-5b0c92d34619" (UID: "5f63ac8a-5b96-47b1-a975-5b0c92d34619"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:00.651936 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651926    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.651988 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.651940    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.653471 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.653450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f63ac8a-5b96-47b1-a975-5b0c92d34619-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f63ac8a-5b96-47b1-a975-5b0c92d34619" (UID: "5f63ac8a-5b96-47b1-a975-5b0c92d34619"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:00.653550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.653504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-api-access-xkk5z" (OuterVolumeSpecName: "kube-api-access-xkk5z") pod "5f63ac8a-5b96-47b1-a975-5b0c92d34619" (UID: "5f63ac8a-5b96-47b1-a975-5b0c92d34619"). InnerVolumeSpecName "kube-api-access-xkk5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:00.719905 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.719877    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerID="32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210" exitCode=0
Sep 17 18:19:00.720047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.719883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerDied","Data":"32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210"}
Sep 17 18:19:00.720047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.720006    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"
Sep 17 18:19:00.720047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.720028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488" event={"ID":"5f63ac8a-5b96-47b1-a975-5b0c92d34619","Type":"ContainerDied","Data":"9fef271b77f94528081fc3037530a84b561bbd6479c4bcd7d0e30b8a4c107ec9"}
Sep 17 18:19:00.720177 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.720059    2563 scope.go:117] "RemoveContainer" containerID="94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445"
Sep 17 18:19:00.729393 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.729374    2563 scope.go:117] "RemoveContainer" containerID="32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210"
Sep 17 18:19:00.736995 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.736981    2563 scope.go:117] "RemoveContainer" containerID="cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6"
Sep 17 18:19:00.744425 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.744403    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"]
Sep 17 18:19:00.744669 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.744649    2563 scope.go:117] "RemoveContainer" containerID="94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445"
Sep 17 18:19:00.744964 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:19:00.744946    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445\": container with ID starting with 94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445 not found: ID does not exist" containerID="94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445"
Sep 17 18:19:00.745050 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.744974    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445"} err="failed to get container status \"94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445\": rpc error: code = NotFound desc = could not find container \"94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445\": container with ID starting with 94f12594e15cc8c69eda30598b91938fa2554c71136bc21a1489d68046399445 not found: ID does not exist"
Sep 17 18:19:00.745050 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.745013    2563 scope.go:117] "RemoveContainer" containerID="32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210"
Sep 17 18:19:00.745284 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:19:00.745267    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210\": container with ID starting with 32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210 not found: ID does not exist" containerID="32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210"
Sep 17 18:19:00.745334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.745289    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210"} err="failed to get container status \"32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210\": rpc error: code = NotFound desc = could not find container \"32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210\": container with ID starting with 32324e64785b270c2d9e02a540b846a9bcfcfe41809fb7fb15e99e656e5d4210 not found: ID does not exist"
Sep 17 18:19:00.745334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.745304    2563 scope.go:117] "RemoveContainer" containerID="cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6"
Sep 17 18:19:00.745501 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:19:00.745479    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6\": container with ID starting with cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6 not found: ID does not exist" containerID="cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6"
Sep 17 18:19:00.745542 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.745512    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6"} err="failed to get container status \"cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6\": rpc error: code = NotFound desc = could not find container \"cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6\": container with ID starting with cdcfd64d52a827727a74257e0052f05233857df06810935af5ecfbb5a1bd8ed6 not found: ID does not exist"
Sep 17 18:19:00.750399 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.750378    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-6v488"]
Sep 17 18:19:00.752801 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.752771    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.752847 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.752814    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkk5z\" (UniqueName: \"kubernetes.io/projected/5f63ac8a-5b96-47b1-a975-5b0c92d34619-kube-api-access-xkk5z\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.752847 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.752826    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f63ac8a-5b96-47b1-a975-5b0c92d34619-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.959152 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:19:00.959119    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" path="/var/lib/kubelet/pods/5f63ac8a-5b96-47b1-a975-5b0c92d34619/volumes"
Sep 17 18:20:21.415631 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:21.415538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.571499 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.571426    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.602572 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"]
Sep 17 18:20:26.602933 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602920    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602934    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="storage-initializer"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602954    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="storage-initializer"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602964    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kube-rbac-proxy"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kube-rbac-proxy"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602983    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="storage-initializer"
Sep 17 18:20:26.602987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602988    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="storage-initializer"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.602999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kserve-container"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603004    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kserve-container"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603016    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kube-rbac-proxy"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603021    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kube-rbac-proxy"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603076    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kube-rbac-proxy"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c489fc4-a269-45ef-bf18-ece988a0b6cc" containerName="kserve-container"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603091    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kube-rbac-proxy"
Sep 17 18:20:26.603204 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.603097    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f63ac8a-5b96-47b1-a975-5b0c92d34619" containerName="kserve-container"
Sep 17 18:20:26.606399 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.606381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.608947 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.608920    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:20:26.609080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.608967    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 18:20:26.609080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.608987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 18:20:26.609080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.609000    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:20:26.609080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.609023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:20:26.609342 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.609326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:20:26.617275 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.617245    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"]
Sep 17 18:20:26.689081 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.689041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.689081 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.689086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.689309 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.689111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.689309 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.689206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plh8q\" (UniqueName: \"kubernetes.io/projected/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-api-access-plh8q\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.790394 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.790354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plh8q\" (UniqueName: \"kubernetes.io/projected/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-api-access-plh8q\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.790579 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.790459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.790579 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.790484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.790579 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.790504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.790706 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:20:26.790627    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 18:20:26.790744 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:20:26.790715    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls podName:ec09e8ae-d2d2-463b-ad2d-194c468002fc nodeName:}" failed. No retries permitted until 2026-09-17 18:20:27.290693124 +0000 UTC m=+2712.818474943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls") pod "isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" (UID: "ec09e8ae-d2d2-463b-ad2d-194c468002fc") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 18:20:26.791011 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.790988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.791241 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.791225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.799398 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.799371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plh8q\" (UniqueName: \"kubernetes.io/projected/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-api-access-plh8q\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:26.939963 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:26.939926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:27.295300 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:27.295205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:27.297642 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:27.297611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls\") pod \"isvc-sklearn-s3-predictor-58d9d56db6-4hsjc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:27.517827 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:27.517777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:27.790191 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:27.790161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"]
Sep 17 18:20:27.792211 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:20:27.792175    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec09e8ae_d2d2_463b_ad2d_194c468002fc.slice/crio-2e9555e6606dc564a62440a23ab29985bd653189e7c68030e11535c787c1bb62 WatchSource:0}: Error finding container 2e9555e6606dc564a62440a23ab29985bd653189e7c68030e11535c787c1bb62: Status 404 returned error can't find the container with id 2e9555e6606dc564a62440a23ab29985bd653189e7c68030e11535c787c1bb62
Sep 17 18:20:28.138921 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:28.138840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerStarted","Data":"138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3"}
Sep 17 18:20:28.138921 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:28.138901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerStarted","Data":"2e9555e6606dc564a62440a23ab29985bd653189e7c68030e11535c787c1bb62"}
Sep 17 18:20:29.144755 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:29.144724    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerID="138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3" exitCode=0
Sep 17 18:20:29.145128 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:29.144816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerDied","Data":"138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3"}
Sep 17 18:20:30.152042 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:30.152012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerStarted","Data":"24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db"}
Sep 17 18:20:30.152435 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:30.152066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerStarted","Data":"954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b"}
Sep 17 18:20:30.152435 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:30.152129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:30.170241 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:30.170191    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podStartSLOduration=2.932466737 podStartE2EDuration="4.170175096s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.237708359s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:30.169496275 +0000 UTC m=+2715.697278100" watchObservedRunningTime="2026-09-17 18:20:30.170175096 +0000 UTC m=+2715.697956923"
Sep 17 18:20:31.156718 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:31.156685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:31.158062 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:31.158032    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:20:32.162131 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:32.162090    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:20:33.364254 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:33.364161    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:20:33.370976 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:33.370943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:20:37.166456 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:37.166426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:20:37.166951 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:37.166924    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:20:47.166892 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:47.166847    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:20:51.321719 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:51.321679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:51.569561 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:51.569534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:56.417482 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.417451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:56.458171 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.458132    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"]
Sep 17 18:20:56.461050 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.461031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.463446 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.463423    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:20:56.463571 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.463433    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:56.463571 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.463514    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:20:56.463821 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.463806    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 18:20:56.476402 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.476381    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"]
Sep 17 18:20:56.544728 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.544698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.544917 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.544736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66ws\" (UniqueName: \"kubernetes.io/projected/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-api-access-m66ws\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.544917 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.544754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.544917 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.544868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.544917 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.544894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.645498 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.645465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.645498 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.645502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.645749 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.645538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.645749 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.645568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66ws\" (UniqueName: \"kubernetes.io/projected/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-api-access-m66ws\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.645749 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.645592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.645948 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.645923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.646347 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.646326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.646386 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.646326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.648001 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.647980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.654249 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.654225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66ws\" (UniqueName: \"kubernetes.io/projected/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-api-access-m66ws\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:56.693705 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.693680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:56.771843 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:56.771819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:20:57.046630 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:57.046601    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"]
Sep 17 18:20:57.048304 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:20:57.048281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5268d5_58b4_45d3_aa2e_a3fc5a6210f9.slice/crio-af59798b5ca34182f50be9036bd32087d069f0555e72ed3b936aaf1ee866a4a4 WatchSource:0}: Error finding container af59798b5ca34182f50be9036bd32087d069f0555e72ed3b936aaf1ee866a4a4: Status 404 returned error can't find the container with id af59798b5ca34182f50be9036bd32087d069f0555e72ed3b936aaf1ee866a4a4
Sep 17 18:20:57.167410 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:57.167369    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:20:57.288719 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:57.288682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerStarted","Data":"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"}
Sep 17 18:20:57.288893 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:20:57.288761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerStarted","Data":"af59798b5ca34182f50be9036bd32087d069f0555e72ed3b936aaf1ee866a4a4"}
Sep 17 18:21:01.309991 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:01.309906    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerID="b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd" exitCode=0
Sep 17 18:21:01.309991 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:01.309953    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerDied","Data":"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"}
Sep 17 18:21:02.317414 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:02.317380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerStarted","Data":"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"}
Sep 17 18:21:02.317897 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:02.317470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerStarted","Data":"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"}
Sep 17 18:21:02.317897 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:02.317669    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:21:03.322625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:03.322587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:21:07.167055 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:07.167017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:21:09.332713 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:09.332682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:21:09.353518 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:09.353469    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" podStartSLOduration=9.453850738 podStartE2EDuration="13.353454102s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899603364s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:02.34013287 +0000 UTC m=+2747.867914696" watchObservedRunningTime="2026-09-17 18:21:09.353454102 +0000 UTC m=+2754.881235927"
Sep 17 18:21:17.167601 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:17.167565    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:21:21.629076 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:21.629011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:27.166897 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:27.166856    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:21:37.167657 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:37.167574    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:21:39.338672 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:39.338637    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:21:46.527572 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:46.527542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"]
Sep 17 18:21:46.528005 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:46.527784    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kserve-container" containerID="cri-o://1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f" gracePeriod=30
Sep 17 18:21:46.528005 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:46.527870    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kube-rbac-proxy" containerID="cri-o://5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a" gracePeriod=30
Sep 17 18:21:46.542566 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:46.542536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:47.168211 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.168177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:21:47.480137 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.480114    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:21:47.532342 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532310    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerID="5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a" exitCode=2
Sep 17 18:21:47.532342 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532334    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerID="1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f" exitCode=0
Sep 17 18:21:47.532823 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532366    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"
Sep 17 18:21:47.532823 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerDied","Data":"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"}
Sep 17 18:21:47.532823 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerDied","Data":"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"}
Sep 17 18:21:47.532823 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2" event={"ID":"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9","Type":"ContainerDied","Data":"af59798b5ca34182f50be9036bd32087d069f0555e72ed3b936aaf1ee866a4a4"}
Sep 17 18:21:47.532823 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.532503    2563 scope.go:117] "RemoveContainer" containerID="5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"
Sep 17 18:21:47.540961 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.540934    2563 scope.go:117] "RemoveContainer" containerID="1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"
Sep 17 18:21:47.548574 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.548556    2563 scope.go:117] "RemoveContainer" containerID="b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"
Sep 17 18:21:47.555527 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.555505    2563 scope.go:117] "RemoveContainer" containerID="5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"
Sep 17 18:21:47.555810 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:21:47.555768    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a\": container with ID starting with 5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a not found: ID does not exist" containerID="5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"
Sep 17 18:21:47.555863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.555816    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"} err="failed to get container status \"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a\": rpc error: code = NotFound desc = could not find container \"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a\": container with ID starting with 5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a not found: ID does not exist"
Sep 17 18:21:47.555863 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.555835    2563 scope.go:117] "RemoveContainer" containerID="1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"
Sep 17 18:21:47.556083 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:21:47.556065    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f\": container with ID starting with 1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f not found: ID does not exist" containerID="1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"
Sep 17 18:21:47.556149 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556093    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"} err="failed to get container status \"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f\": rpc error: code = NotFound desc = could not find container \"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f\": container with ID starting with 1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f not found: ID does not exist"
Sep 17 18:21:47.556149 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556115    2563 scope.go:117] "RemoveContainer" containerID="b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"
Sep 17 18:21:47.556384 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:21:47.556361    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd\": container with ID starting with b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd not found: ID does not exist" containerID="b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"
Sep 17 18:21:47.556435 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556384    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"} err="failed to get container status \"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd\": rpc error: code = NotFound desc = could not find container \"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd\": container with ID starting with b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd not found: ID does not exist"
Sep 17 18:21:47.556435 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556397    2563 scope.go:117] "RemoveContainer" containerID="5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"
Sep 17 18:21:47.556629 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556613    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a"} err="failed to get container status \"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a\": rpc error: code = NotFound desc = could not find container \"5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a\": container with ID starting with 5d14d889aeacc572fd5db348e4f5f471dd13a01d2ced9863c656bc3113d4573a not found: ID does not exist"
Sep 17 18:21:47.556676 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556628    2563 scope.go:117] "RemoveContainer" containerID="1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"
Sep 17 18:21:47.556860 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556837    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f"} err="failed to get container status \"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f\": rpc error: code = NotFound desc = could not find container \"1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f\": container with ID starting with 1636d5bf5e4aa7840d0f9acfbf511aa6ee231cd9766c7984dbab278bfe290d5f not found: ID does not exist"
Sep 17 18:21:47.556860 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.556861    2563 scope.go:117] "RemoveContainer" containerID="b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"
Sep 17 18:21:47.557098 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.557078    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd"} err="failed to get container status \"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd\": rpc error: code = NotFound desc = could not find container \"b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd\": container with ID starting with b5fcf01bc70b78fbcfea82494767e483bcefaae4d22d8624984437d09b3b06fd not found: ID does not exist"
Sep 17 18:21:47.583719 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.583699    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-rbac-proxy-sar-config\") pod \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") "
Sep 17 18:21:47.583826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.583736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m66ws\" (UniqueName: \"kubernetes.io/projected/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-api-access-m66ws\") pod \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") "
Sep 17 18:21:47.583826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.583809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-proxy-tls\") pod \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") "
Sep 17 18:21:47.583954 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.583851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-cabundle-cert\") pod \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") "
Sep 17 18:21:47.584033 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.584009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kserve-provision-location\") pod \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\" (UID: \"0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9\") "
Sep 17 18:21:47.584151 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.584013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" (UID: "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:47.584280 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.584252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" (UID: "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:47.584383 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.584290    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:21:47.584383 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.584354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" (UID: "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:47.586147 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.586123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-api-access-m66ws" (OuterVolumeSpecName: "kube-api-access-m66ws") pod "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" (UID: "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9"). InnerVolumeSpecName "kube-api-access-m66ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:47.586147 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.586124    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" (UID: "0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:47.685321 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.685299    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:21:47.685321 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.685317    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:21:47.685453 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.685327    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:21:47.685453 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.685335    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m66ws\" (UniqueName: \"kubernetes.io/projected/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9-kube-api-access-m66ws\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:21:47.855885 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.855854    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"]
Sep 17 18:21:47.861893 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:47.861867    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cnvw2"]
Sep 17 18:21:48.958572 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:48.958536    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" path="/var/lib/kubelet/pods/0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9/volumes"
Sep 17 18:21:51.686620 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:51.686586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:51.925470 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:51.925436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:56.678165 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:56.678133    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"]
Sep 17 18:21:56.678625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:56.678413    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" containerID="cri-o://954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b" gracePeriod=30
Sep 17 18:21:56.678625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:56.678455    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kube-rbac-proxy" containerID="cri-o://24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db" gracePeriod=30
Sep 17 18:21:56.704762 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:56.704740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:56.820295 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:56.820258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:57.162426 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:57.162342    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Sep 17 18:21:57.166855 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:57.166830    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Sep 17 18:21:57.192889 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:57.192857    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:57.589932 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:57.589896    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerID="24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db" exitCode=2
Sep 17 18:21:57.590106 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:21:57.589968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerDied","Data":"24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db"}
Sep 17 18:22:01.426366 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.426341    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:22:01.500508 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.500415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls\") pod \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") "
Sep 17 18:22:01.500508 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.500484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-rbac-proxy-sar-config\") pod \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") "
Sep 17 18:22:01.500750 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.500560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kserve-provision-location\") pod \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") "
Sep 17 18:22:01.500750 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.500582    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plh8q\" (UniqueName: \"kubernetes.io/projected/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-api-access-plh8q\") pod \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\" (UID: \"ec09e8ae-d2d2-463b-ad2d-194c468002fc\") "
Sep 17 18:22:01.500885 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.500867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec09e8ae-d2d2-463b-ad2d-194c468002fc" (UID: "ec09e8ae-d2d2-463b-ad2d-194c468002fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:01.500925 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.500884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec09e8ae-d2d2-463b-ad2d-194c468002fc" (UID: "ec09e8ae-d2d2-463b-ad2d-194c468002fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:01.502564 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.502538    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec09e8ae-d2d2-463b-ad2d-194c468002fc" (UID: "ec09e8ae-d2d2-463b-ad2d-194c468002fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:01.502660 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.502571    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-api-access-plh8q" (OuterVolumeSpecName: "kube-api-access-plh8q") pod "ec09e8ae-d2d2-463b-ad2d-194c468002fc" (UID: "ec09e8ae-d2d2-463b-ad2d-194c468002fc"). InnerVolumeSpecName "kube-api-access-plh8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:01.601304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.601273    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:22:01.601304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.601299    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plh8q\" (UniqueName: \"kubernetes.io/projected/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-api-access-plh8q\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:22:01.601304 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.601309    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec09e8ae-d2d2-463b-ad2d-194c468002fc-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:22:01.601529 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.601318    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec09e8ae-d2d2-463b-ad2d-194c468002fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:22:01.612463 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.612429    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerID="954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b" exitCode=0
Sep 17 18:22:01.612629 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.612479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerDied","Data":"954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b"}
Sep 17 18:22:01.612629 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.612491    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"
Sep 17 18:22:01.612629 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.612530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc" event={"ID":"ec09e8ae-d2d2-463b-ad2d-194c468002fc","Type":"ContainerDied","Data":"2e9555e6606dc564a62440a23ab29985bd653189e7c68030e11535c787c1bb62"}
Sep 17 18:22:01.612629 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.612578    2563 scope.go:117] "RemoveContainer" containerID="24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db"
Sep 17 18:22:01.622024 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.622005    2563 scope.go:117] "RemoveContainer" containerID="954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b"
Sep 17 18:22:01.629845 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.629822    2563 scope.go:117] "RemoveContainer" containerID="138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3"
Sep 17 18:22:01.634834 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.634811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"]
Sep 17 18:22:01.637873 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.637848    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-58d9d56db6-4hsjc"]
Sep 17 18:22:01.638258 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.638235    2563 scope.go:117] "RemoveContainer" containerID="24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db"
Sep 17 18:22:01.638549 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:22:01.638531    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db\": container with ID starting with 24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db not found: ID does not exist" containerID="24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db"
Sep 17 18:22:01.638612 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.638559    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db"} err="failed to get container status \"24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db\": rpc error: code = NotFound desc = could not find container \"24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db\": container with ID starting with 24ed9d8202049022a11b90af3206d2bee32ca3035c6c1df6a5a8b466582cf8db not found: ID does not exist"
Sep 17 18:22:01.638612 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.638579    2563 scope.go:117] "RemoveContainer" containerID="954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b"
Sep 17 18:22:01.638878 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:22:01.638860    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b\": container with ID starting with 954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b not found: ID does not exist" containerID="954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b"
Sep 17 18:22:01.638953 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.638888    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b"} err="failed to get container status \"954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b\": rpc error: code = NotFound desc = could not find container \"954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b\": container with ID starting with 954035c233a311d47e42aacb75401658140aa6ea5d2fda8e97fcb7b05168bb1b not found: ID does not exist"
Sep 17 18:22:01.638953 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.638910    2563 scope.go:117] "RemoveContainer" containerID="138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3"
Sep 17 18:22:01.639177 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:22:01.639160    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3\": container with ID starting with 138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3 not found: ID does not exist" containerID="138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3"
Sep 17 18:22:01.639226 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:01.639182    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3"} err="failed to get container status \"138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3\": rpc error: code = NotFound desc = could not find container \"138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3\": container with ID starting with 138b5be80de578a98847e7fb837b219d5bea6bdf47e8939ccbd7c2d3ec852fd3 not found: ID does not exist"
Sep 17 18:22:02.958377 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:02.958344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" path="/var/lib/kubelet/pods/ec09e8ae-d2d2-463b-ad2d-194c468002fc/volumes"
Sep 17 18:22:21.840550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:21.840515    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:26.996080 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:26.996042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:27.037699 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.037670    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"]
Sep 17 18:22:27.038086 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038068    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kube-rbac-proxy"
Sep 17 18:22:27.038086 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038085    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kube-rbac-proxy"
Sep 17 18:22:27.038086 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038097    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kserve-container"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038107    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kserve-container"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="storage-initializer"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038127    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="storage-initializer"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038135    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038154    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kube-rbac-proxy"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038159    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kube-rbac-proxy"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038188    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="storage-initializer"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038196    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="storage-initializer"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038265    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kube-rbac-proxy"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038275    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kube-rbac-proxy"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038282    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5268d5-58b4-45d3-aa2e-a3fc5a6210f9" containerName="kserve-container"
Sep 17 18:22:27.038317 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.038288    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec09e8ae-d2d2-463b-ad2d-194c468002fc" containerName="kserve-container"
Sep 17 18:22:27.041449 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.041432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.044585 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.044564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:22:27.045201 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.045182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:22:27.045340 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.045181    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:22:27.045340 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.045184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:22:27.045536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.045189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:27.045536 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.045228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 18:22:27.045821 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.045781    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:27.054905 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.054887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"]
Sep 17 18:22:27.123517 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.123482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d634d671-4f13-4968-8bb0-a38e1e24af12-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.123661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.123521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjwjp\" (UniqueName: \"kubernetes.io/projected/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-api-access-rjwjp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.123661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.123550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.123754 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.123664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.123754 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.123743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d634d671-4f13-4968-8bb0-a38e1e24af12-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.202261 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.202231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:27.224335 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.224312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.224419 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.224356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d634d671-4f13-4968-8bb0-a38e1e24af12-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.224419 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.224400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d634d671-4f13-4968-8bb0-a38e1e24af12-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.224419 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.224416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjwjp\" (UniqueName: \"kubernetes.io/projected/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-api-access-rjwjp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.224560 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.224431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.224904 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.224878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d634d671-4f13-4968-8bb0-a38e1e24af12-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.225098 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.225078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.225167 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.225081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.226824 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.226784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d634d671-4f13-4968-8bb0-a38e1e24af12-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.233736 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.233718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjwjp\" (UniqueName: \"kubernetes.io/projected/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-api-access-rjwjp\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.352144 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.352050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:27.630447 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.630421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"]
Sep 17 18:22:27.632239 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:22:27.632195    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd634d671_4f13_4968_8bb0_a38e1e24af12.slice/crio-691b4363079e477643219cf7cb4f7b7723cd1991d0c2389fb678b319bfbe9789 WatchSource:0}: Error finding container 691b4363079e477643219cf7cb4f7b7723cd1991d0c2389fb678b319bfbe9789: Status 404 returned error can't find the container with id 691b4363079e477643219cf7cb4f7b7723cd1991d0c2389fb678b319bfbe9789
Sep 17 18:22:27.738711 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.738678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerStarted","Data":"466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9"}
Sep 17 18:22:27.738886 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:27.738742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerStarted","Data":"691b4363079e477643219cf7cb4f7b7723cd1991d0c2389fb678b319bfbe9789"}
Sep 17 18:22:31.757987 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:31.757949    2563 generic.go:356] "Generic (PLEG): container finished" podID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerID="466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9" exitCode=0
Sep 17 18:22:31.758424 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:31.758024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerDied","Data":"466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9"}
Sep 17 18:22:32.765152 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:32.765119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerStarted","Data":"b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd"}
Sep 17 18:22:32.765548 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:32.765182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerStarted","Data":"53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1"}
Sep 17 18:22:32.765548 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:32.765379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:32.790734 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:32.790683    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" podStartSLOduration=1.9180860229999999 podStartE2EDuration="5.790666356s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.872580333s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:32.788942779 +0000 UTC m=+2838.316724606" watchObservedRunningTime="2026-09-17 18:22:32.790666356 +0000 UTC m=+2838.318448182"
Sep 17 18:22:33.770701 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:33.770665    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:22:39.781690 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:22:39.781662    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:23:09.786627 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:09.786553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:23:17.114715 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:17.114682    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"]
Sep 17 18:23:17.115132 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:17.114939    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kserve-container" containerID="cri-o://53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1" gracePeriod=30
Sep 17 18:23:17.115132 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:17.114986    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kube-rbac-proxy" containerID="cri-o://b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd" gracePeriod=30
Sep 17 18:23:17.146805 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:17.146754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:17.978588 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:17.978546    2563 generic.go:356] "Generic (PLEG): container finished" podID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerID="b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd" exitCode=2
Sep 17 18:23:17.978769 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:17.978615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerDied","Data":"b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd"}
Sep 17 18:23:18.364601 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.364576    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:23:18.480989 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.480958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-rbac-proxy-sar-config\") pod \"d634d671-4f13-4968-8bb0-a38e1e24af12\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") "
Sep 17 18:23:18.481162 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.480997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-cabundle-cert\") pod \"d634d671-4f13-4968-8bb0-a38e1e24af12\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") "
Sep 17 18:23:18.481162 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.481020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjwjp\" (UniqueName: \"kubernetes.io/projected/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-api-access-rjwjp\") pod \"d634d671-4f13-4968-8bb0-a38e1e24af12\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") "
Sep 17 18:23:18.481162 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.481045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d634d671-4f13-4968-8bb0-a38e1e24af12-kserve-provision-location\") pod \"d634d671-4f13-4968-8bb0-a38e1e24af12\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") "
Sep 17 18:23:18.481162 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.481096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d634d671-4f13-4968-8bb0-a38e1e24af12-proxy-tls\") pod \"d634d671-4f13-4968-8bb0-a38e1e24af12\" (UID: \"d634d671-4f13-4968-8bb0-a38e1e24af12\") "
Sep 17 18:23:18.481390 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.481363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d634d671-4f13-4968-8bb0-a38e1e24af12" (UID: "d634d671-4f13-4968-8bb0-a38e1e24af12"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:18.481448 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.481382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634d671-4f13-4968-8bb0-a38e1e24af12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d634d671-4f13-4968-8bb0-a38e1e24af12" (UID: "d634d671-4f13-4968-8bb0-a38e1e24af12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:18.481448 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.481421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d634d671-4f13-4968-8bb0-a38e1e24af12" (UID: "d634d671-4f13-4968-8bb0-a38e1e24af12"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:18.483170 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.483114    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-api-access-rjwjp" (OuterVolumeSpecName: "kube-api-access-rjwjp") pod "d634d671-4f13-4968-8bb0-a38e1e24af12" (UID: "d634d671-4f13-4968-8bb0-a38e1e24af12"). InnerVolumeSpecName "kube-api-access-rjwjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:18.483170 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.483144    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d634d671-4f13-4968-8bb0-a38e1e24af12-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d634d671-4f13-4968-8bb0-a38e1e24af12" (UID: "d634d671-4f13-4968-8bb0-a38e1e24af12"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:18.582193 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.582166    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:18.582193 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.582188    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d634d671-4f13-4968-8bb0-a38e1e24af12-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:18.582193 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.582198    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjwjp\" (UniqueName: \"kubernetes.io/projected/d634d671-4f13-4968-8bb0-a38e1e24af12-kube-api-access-rjwjp\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:18.582364 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.582207    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d634d671-4f13-4968-8bb0-a38e1e24af12-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:18.582364 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.582216    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d634d671-4f13-4968-8bb0-a38e1e24af12-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:18.984911 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.984875    2563 generic.go:356] "Generic (PLEG): container finished" podID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerID="53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1" exitCode=0
Sep 17 18:23:18.985058 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.984937    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"
Sep 17 18:23:18.985058 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.984941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerDied","Data":"53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1"}
Sep 17 18:23:18.985171 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.985073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5" event={"ID":"d634d671-4f13-4968-8bb0-a38e1e24af12","Type":"ContainerDied","Data":"691b4363079e477643219cf7cb4f7b7723cd1991d0c2389fb678b319bfbe9789"}
Sep 17 18:23:18.985171 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.985102    2563 scope.go:117] "RemoveContainer" containerID="b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd"
Sep 17 18:23:18.993433 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:18.993415    2563 scope.go:117] "RemoveContainer" containerID="53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1"
Sep 17 18:23:19.001163 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.001102    2563 scope.go:117] "RemoveContainer" containerID="466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9"
Sep 17 18:23:19.003542 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.003523    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"]
Sep 17 18:23:19.007580 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.007556    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-66rj5"]
Sep 17 18:23:19.009462 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.009448    2563 scope.go:117] "RemoveContainer" containerID="b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd"
Sep 17 18:23:19.009705 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:19.009686    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd\": container with ID starting with b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd not found: ID does not exist" containerID="b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd"
Sep 17 18:23:19.009773 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.009716    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd"} err="failed to get container status \"b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd\": rpc error: code = NotFound desc = could not find container \"b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd\": container with ID starting with b6a1bd3ed665c009d4f176437b122bf4d80c221ff8bce07478c7d19be70bb5cd not found: ID does not exist"
Sep 17 18:23:19.009773 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.009737    2563 scope.go:117] "RemoveContainer" containerID="53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1"
Sep 17 18:23:19.009982 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:19.009965    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1\": container with ID starting with 53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1 not found: ID does not exist" containerID="53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1"
Sep 17 18:23:19.010026 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.009988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1"} err="failed to get container status \"53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1\": rpc error: code = NotFound desc = could not find container \"53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1\": container with ID starting with 53002103c22d389838462d7e9c36ec58852d6d77011fd3f0e6343b80d9d1bed1 not found: ID does not exist"
Sep 17 18:23:19.010026 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.010002    2563 scope.go:117] "RemoveContainer" containerID="466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9"
Sep 17 18:23:19.010244 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:19.010230    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9\": container with ID starting with 466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9 not found: ID does not exist" containerID="466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9"
Sep 17 18:23:19.010286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:19.010247    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9"} err="failed to get container status \"466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9\": rpc error: code = NotFound desc = could not find container \"466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9\": container with ID starting with 466639865e97a8504f7f9230dbe056a19d855675337716dad11f68a3037578c9 not found: ID does not exist"
Sep 17 18:23:20.958512 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:20.958482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" path="/var/lib/kubelet/pods/d634d671-4f13-4968-8bb0-a38e1e24af12/volumes"
Sep 17 18:23:22.401498 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:22.401462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:22.501566 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:22.501538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:26.876394 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:26.876359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:27.963175 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.963143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:27.990107 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990070    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"]
Sep 17 18:23:27.990612 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990598    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kserve-container"
Sep 17 18:23:27.990656 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990616    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kserve-container"
Sep 17 18:23:27.990656 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kube-rbac-proxy"
Sep 17 18:23:27.990656 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990637    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kube-rbac-proxy"
Sep 17 18:23:27.990753 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990667    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="storage-initializer"
Sep 17 18:23:27.990753 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="storage-initializer"
Sep 17 18:23:27.990833 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990758    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kserve-container"
Sep 17 18:23:27.990833 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.990767    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d634d671-4f13-4968-8bb0-a38e1e24af12" containerName="kube-rbac-proxy"
Sep 17 18:23:27.994087 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.994059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:27.996450 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.996426    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:23:27.996564 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.996537    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:23:27.996625 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.996609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 18:23:27.996681 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.996648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:23:27.996735 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.996688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:23:27.996812 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:27.996753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:23:28.006465 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.006440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"]
Sep 17 18:23:28.061277 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.061249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.061415 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.061300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51f0767e-919b-4967-892c-bf4f2bc15894-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.061415 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.061399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51f0767e-919b-4967-892c-bf4f2bc15894-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.061496 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.061426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfx4t\" (UniqueName: \"kubernetes.io/projected/51f0767e-919b-4967-892c-bf4f2bc15894-kube-api-access-jfx4t\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.162069 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.162038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51f0767e-919b-4967-892c-bf4f2bc15894-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.162194 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.162112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51f0767e-919b-4967-892c-bf4f2bc15894-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.162194 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.162139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfx4t\" (UniqueName: \"kubernetes.io/projected/51f0767e-919b-4967-892c-bf4f2bc15894-kube-api-access-jfx4t\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.162324 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.162204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.162377 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:28.162320    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 17 18:23:28.162433 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:28.162389    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls podName:51f0767e-919b-4967-892c-bf4f2bc15894 nodeName:}" failed. No retries permitted until 2026-09-17 18:23:28.662368686 +0000 UTC m=+2894.190150495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" (UID: "51f0767e-919b-4967-892c-bf4f2bc15894") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Sep 17 18:23:28.162501 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.162439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51f0767e-919b-4967-892c-bf4f2bc15894-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.162906 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.162884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51f0767e-919b-4967-892c-bf4f2bc15894-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.170667 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.170643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfx4t\" (UniqueName: \"kubernetes.io/projected/51f0767e-919b-4967-892c-bf4f2bc15894-kube-api-access-jfx4t\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.262518 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.262436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:28.665046 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.664969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.667381 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.667350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:28.906703 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:28.906661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:29.180723 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:29.180696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"]
Sep 17 18:23:29.183669 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:23:29.183634    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f0767e_919b_4967_892c_bf4f2bc15894.slice/crio-d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095 WatchSource:0}: Error finding container d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095: Status 404 returned error can't find the container with id d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095
Sep 17 18:23:30.036868 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:30.036821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" event={"ID":"51f0767e-919b-4967-892c-bf4f2bc15894","Type":"ContainerStarted","Data":"96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0"}
Sep 17 18:23:30.037047 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:30.036903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" event={"ID":"51f0767e-919b-4967-892c-bf4f2bc15894","Type":"ContainerStarted","Data":"d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095"}
Sep 17 18:23:32.048972 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:32.048934    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/0.log"
Sep 17 18:23:32.049434 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:32.048976    2563 generic.go:356] "Generic (PLEG): container finished" podID="51f0767e-919b-4967-892c-bf4f2bc15894" containerID="96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0" exitCode=1
Sep 17 18:23:32.049434 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:32.049036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" event={"ID":"51f0767e-919b-4967-892c-bf4f2bc15894","Type":"ContainerDied","Data":"96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0"}
Sep 17 18:23:33.054802 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:33.054777    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/0.log"
Sep 17 18:23:33.055169 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:33.054869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" event={"ID":"51f0767e-919b-4967-892c-bf4f2bc15894","Type":"ContainerStarted","Data":"4bc627b4e4bc4c7ff0004832122fbd1d56cf26d60fb63058b474306066862245"}
Sep 17 18:23:36.069588 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.069558    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/1.log"
Sep 17 18:23:36.070017 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.069946    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/0.log"
Sep 17 18:23:36.070017 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.069978    2563 generic.go:356] "Generic (PLEG): container finished" podID="51f0767e-919b-4967-892c-bf4f2bc15894" containerID="4bc627b4e4bc4c7ff0004832122fbd1d56cf26d60fb63058b474306066862245" exitCode=1
Sep 17 18:23:36.070091 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.070052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" event={"ID":"51f0767e-919b-4967-892c-bf4f2bc15894","Type":"ContainerDied","Data":"4bc627b4e4bc4c7ff0004832122fbd1d56cf26d60fb63058b474306066862245"}
Sep 17 18:23:36.070123 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.070109    2563 scope.go:117] "RemoveContainer" containerID="96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0"
Sep 17 18:23:36.070447 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.070429    2563 scope.go:117] "RemoveContainer" containerID="96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0"
Sep 17 18:23:36.081077 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:36.081057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_kserve-ci-e2e-test(51f0767e-919b-4967-892c-bf4f2bc15894)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" podUID="51f0767e-919b-4967-892c-bf4f2bc15894"
Sep 17 18:23:36.081163 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:23:36.081105    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_kserve-ci-e2e-test_51f0767e-919b-4967-892c-bf4f2bc15894_0 in pod sandbox d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095 from index: no such id: '96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0'" containerID="96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0"
Sep 17 18:23:36.081163 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:36.081135    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_kserve-ci-e2e-test_51f0767e-919b-4967-892c-bf4f2bc15894_0 in pod sandbox d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095 from index: no such id: '96ed8c1f29e0328d9784d9d5455c4b47ddfedea320b8437e7ab306957c300fa0'"
Sep 17 18:23:37.075956 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:37.075927    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/1.log"
Sep 17 18:23:37.951572 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:37.951537    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"]
Sep 17 18:23:37.973680 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:37.973644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:38.077459 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.077439    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/1.log"
Sep 17 18:23:38.081067 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.081043    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:38.081935 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.081917    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm_51f0767e-919b-4967-892c-bf4f2bc15894/storage-initializer/1.log"
Sep 17 18:23:38.082076 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.081980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm" event={"ID":"51f0767e-919b-4967-892c-bf4f2bc15894","Type":"ContainerDied","Data":"d28b20f0dc5af88724069fc2fe3afce80e04faa7fad86b1c5d80d2b2fd79f095"}
Sep 17 18:23:38.082076 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.082034    2563 scope.go:117] "RemoveContainer" containerID="4bc627b4e4bc4c7ff0004832122fbd1d56cf26d60fb63058b474306066862245"
Sep 17 18:23:38.140826 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.140768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51f0767e-919b-4967-892c-bf4f2bc15894-kube-rbac-proxy-sar-config\") pod \"51f0767e-919b-4967-892c-bf4f2bc15894\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") "
Sep 17 18:23:38.140939 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.140862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51f0767e-919b-4967-892c-bf4f2bc15894-kserve-provision-location\") pod \"51f0767e-919b-4967-892c-bf4f2bc15894\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") "
Sep 17 18:23:38.140939 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.140894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls\") pod \"51f0767e-919b-4967-892c-bf4f2bc15894\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") "
Sep 17 18:23:38.140939 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.140911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfx4t\" (UniqueName: \"kubernetes.io/projected/51f0767e-919b-4967-892c-bf4f2bc15894-kube-api-access-jfx4t\") pod \"51f0767e-919b-4967-892c-bf4f2bc15894\" (UID: \"51f0767e-919b-4967-892c-bf4f2bc15894\") "
Sep 17 18:23:38.141112 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.141090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f0767e-919b-4967-892c-bf4f2bc15894-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51f0767e-919b-4967-892c-bf4f2bc15894" (UID: "51f0767e-919b-4967-892c-bf4f2bc15894"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:38.141154 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.141114    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f0767e-919b-4967-892c-bf4f2bc15894-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51f0767e-919b-4967-892c-bf4f2bc15894" (UID: "51f0767e-919b-4967-892c-bf4f2bc15894"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:38.141252 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.141230    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51f0767e-919b-4967-892c-bf4f2bc15894-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:38.141252 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.141254    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51f0767e-919b-4967-892c-bf4f2bc15894-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:38.142923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.142898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51f0767e-919b-4967-892c-bf4f2bc15894" (UID: "51f0767e-919b-4967-892c-bf4f2bc15894"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:38.142923 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.142908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f0767e-919b-4967-892c-bf4f2bc15894-kube-api-access-jfx4t" (OuterVolumeSpecName: "kube-api-access-jfx4t") pod "51f0767e-919b-4967-892c-bf4f2bc15894" (UID: "51f0767e-919b-4967-892c-bf4f2bc15894"). InnerVolumeSpecName "kube-api-access-jfx4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:38.242213 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.242160    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51f0767e-919b-4967-892c-bf4f2bc15894-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:38.242213 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:38.242180    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfx4t\" (UniqueName: \"kubernetes.io/projected/51f0767e-919b-4967-892c-bf4f2bc15894-kube-api-access-jfx4t\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:23:39.092474 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:39.092286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:39.095004 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:39.094982    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"
Sep 17 18:23:39.138208 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:39.138176    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"]
Sep 17 18:23:39.145767 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:39.145742    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5bcb448694-wdppm"]
Sep 17 18:23:39.550513 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:39.550484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:40.957956 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:23:40.957921    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" path="/var/lib/kubelet/pods/51f0767e-919b-4967-892c-bf4f2bc15894/volumes"
Sep 17 18:24:42.363450 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:42.363375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:47.512615 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.512576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:47.538509 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.538483    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"]
Sep 17 18:24:47.538883 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.538868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" containerName="storage-initializer"
Sep 17 18:24:47.538933 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.538884    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" containerName="storage-initializer"
Sep 17 18:24:47.538968 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.538948    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" containerName="storage-initializer"
Sep 17 18:24:47.538968 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.538956    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" containerName="storage-initializer"
Sep 17 18:24:47.539032 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.539015    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" containerName="storage-initializer"
Sep 17 18:24:47.539032 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.539021    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f0767e-919b-4967-892c-bf4f2bc15894" containerName="storage-initializer"
Sep 17 18:24:47.542254 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.542238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.544722 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.544697    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 18:24:47.544926 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.544900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:24:47.544926 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.544915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:24:47.545190 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.545169    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:24:47.545285 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.545226    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:24:47.545474 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.545459    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 18:24:47.545556 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.545466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:24:47.553366 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.553347    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"]
Sep 17 18:24:47.624979 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.624950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.625170 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.624991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.625170 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.625026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tlk\" (UniqueName: \"kubernetes.io/projected/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-api-access-g2tlk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.625170 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.625067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.625170 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.625111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.726484 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.726446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.726667 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.726496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.726667 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.726522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.726667 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.726561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tlk\" (UniqueName: \"kubernetes.io/projected/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-api-access-g2tlk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.726667 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.726579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.726667 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:24:47.726618    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:24:47.726969 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:24:47.726698    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls podName:925ad428-c9ad-4e66-a9f8-a1bc08255d5b nodeName:}" failed. No retries permitted until 2026-09-17 18:24:48.226675974 +0000 UTC m=+2973.754457778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-mc2nw" (UID: "925ad428-c9ad-4e66-a9f8-a1bc08255d5b") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:24:47.727037 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.727019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.727232 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.727215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.727293 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.727281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:47.737002 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:47.736981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tlk\" (UniqueName: \"kubernetes.io/projected/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-api-access-g2tlk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:48.149584 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:48.149548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:48.232346 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:48.232316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:48.234865 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:48.234844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mc2nw\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:48.453249 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:48.453216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:24:48.727227 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:48.727143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"]
Sep 17 18:24:49.425053 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:49.425015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerStarted","Data":"c5944f4438aba957d5413ba76c5d47d966a2f018bc97009935b71096ede53a23"}
Sep 17 18:24:49.425260 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:49.425082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerStarted","Data":"6a36316069424d89a4b7f9b79422f0c01948c56fd18a90bc8a243853be281ecc"}
Sep 17 18:24:54.450373 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:54.450336    2563 generic.go:356] "Generic (PLEG): container finished" podID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerID="c5944f4438aba957d5413ba76c5d47d966a2f018bc97009935b71096ede53a23" exitCode=0
Sep 17 18:24:54.450761 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:54.450415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerDied","Data":"c5944f4438aba957d5413ba76c5d47d966a2f018bc97009935b71096ede53a23"}
Sep 17 18:24:54.451839 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:24:54.451823    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:25:05.512833 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:05.512805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerStarted","Data":"83e4ebe5743c88c786eab0fd7084e14c76600c05bebc80850b7684b11c5dcfcb"}
Sep 17 18:25:06.519861 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:06.519821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerStarted","Data":"42fc01d9db3c77c35401dece247cf5b01478608158509695d4032e6c562913aa"}
Sep 17 18:25:06.520241 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:06.519890    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:25:06.520241 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:06.519914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:25:06.521632 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:06.521600    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 18:25:06.541552 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:06.541497    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podStartSLOduration=3.485321131 podStartE2EDuration="19.541480653s" totalImagesPullingTime="10.978250977s" totalInitContainerRuntime="5.077908545s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:06.540533621 +0000 UTC m=+2992.068315448" watchObservedRunningTime="2026-09-17 18:25:06.541480653 +0000 UTC m=+2992.069262544"
Sep 17 18:25:07.524707 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:07.524665    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 18:25:09.145562 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:09.145481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:10.237072 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:10.237035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:10.529566 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:10.529479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:12.529690 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:12.529660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:25:12.530187 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:12.530160    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 18:25:20.234326 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:20.234288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:21.344841 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:21.344771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:21.417194 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:21.417160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:22.530341 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:22.530297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 18:25:32.530736 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:32.530693    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 18:25:33.396688 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:33.396576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:25:33.636914 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:33.404235    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:25:42.530524 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:42.530479    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 17 18:25:52.531618 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:52.531589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:25:58.863828 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:58.863776    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"]
Sep 17 18:25:58.864286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:58.864133    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container" containerID="cri-o://83e4ebe5743c88c786eab0fd7084e14c76600c05bebc80850b7684b11c5dcfcb" gracePeriod=30
Sep 17 18:25:58.864286 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:58.864223    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kube-rbac-proxy" containerID="cri-o://42fc01d9db3c77c35401dece247cf5b01478608158509695d4032e6c562913aa" gracePeriod=30
Sep 17 18:25:58.884882 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:58.884849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:59.773460 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:59.773425    2563 generic.go:356] "Generic (PLEG): container finished" podID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerID="42fc01d9db3c77c35401dece247cf5b01478608158509695d4032e6c562913aa" exitCode=2
Sep 17 18:25:59.773460 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:25:59.773465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerDied","Data":"42fc01d9db3c77c35401dece247cf5b01478608158509695d4032e6c562913aa"}
Sep 17 18:26:01.785007 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:01.784971    2563 generic.go:356] "Generic (PLEG): container finished" podID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerID="83e4ebe5743c88c786eab0fd7084e14c76600c05bebc80850b7684b11c5dcfcb" exitCode=0
Sep 17 18:26:01.785321 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:01.785038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerDied","Data":"83e4ebe5743c88c786eab0fd7084e14c76600c05bebc80850b7684b11c5dcfcb"}
Sep 17 18:26:01.909611 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:01.909585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:26:02.065425 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-cabundle-cert\") pod \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") "
Sep 17 18:26:02.065425 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kserve-provision-location\") pod \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") "
Sep 17 18:26:02.065425 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls\") pod \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") "
Sep 17 18:26:02.065425 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-rbac-proxy-sar-config\") pod \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") "
Sep 17 18:26:02.065767 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2tlk\" (UniqueName: \"kubernetes.io/projected/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-api-access-g2tlk\") pod \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\" (UID: \"925ad428-c9ad-4e66-a9f8-a1bc08255d5b\") "
Sep 17 18:26:02.065767 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "925ad428-c9ad-4e66-a9f8-a1bc08255d5b" (UID: "925ad428-c9ad-4e66-a9f8-a1bc08255d5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:02.065920 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.065841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "925ad428-c9ad-4e66-a9f8-a1bc08255d5b" (UID: "925ad428-c9ad-4e66-a9f8-a1bc08255d5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:02.067630 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.067596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "925ad428-c9ad-4e66-a9f8-a1bc08255d5b" (UID: "925ad428-c9ad-4e66-a9f8-a1bc08255d5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:02.067630 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.067602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-api-access-g2tlk" (OuterVolumeSpecName: "kube-api-access-g2tlk") pod "925ad428-c9ad-4e66-a9f8-a1bc08255d5b" (UID: "925ad428-c9ad-4e66-a9f8-a1bc08255d5b"). InnerVolumeSpecName "kube-api-access-g2tlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:02.074926 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.074905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "925ad428-c9ad-4e66-a9f8-a1bc08255d5b" (UID: "925ad428-c9ad-4e66-a9f8-a1bc08255d5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:02.166181 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.166134    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-cabundle-cert\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:26:02.166181 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.166176    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:26:02.166181 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.166191    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:26:02.166181 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.166200    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:26:02.166463 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.166208    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tlk\" (UniqueName: \"kubernetes.io/projected/925ad428-c9ad-4e66-a9f8-a1bc08255d5b-kube-api-access-g2tlk\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:26:02.792094 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.792059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw" event={"ID":"925ad428-c9ad-4e66-a9f8-a1bc08255d5b","Type":"ContainerDied","Data":"6a36316069424d89a4b7f9b79422f0c01948c56fd18a90bc8a243853be281ecc"}
Sep 17 18:26:02.792531 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.792104    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"
Sep 17 18:26:02.792531 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.792127    2563 scope.go:117] "RemoveContainer" containerID="42fc01d9db3c77c35401dece247cf5b01478608158509695d4032e6c562913aa"
Sep 17 18:26:02.801152 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.801136    2563 scope.go:117] "RemoveContainer" containerID="83e4ebe5743c88c786eab0fd7084e14c76600c05bebc80850b7684b11c5dcfcb"
Sep 17 18:26:02.808629 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.808611    2563 scope.go:117] "RemoveContainer" containerID="c5944f4438aba957d5413ba76c5d47d966a2f018bc97009935b71096ede53a23"
Sep 17 18:26:02.814968 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.814946    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"]
Sep 17 18:26:02.818969 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.818947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mc2nw"]
Sep 17 18:26:02.958702 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:02.958666    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" path="/var/lib/kubelet/pods/925ad428-c9ad-4e66-a9f8-a1bc08255d5b/volumes"
Sep 17 18:26:04.027055 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:04.027022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:04.379226 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:04.379141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.395965 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:51.395929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:52.473914 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.473873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:52.495949 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.495911    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"]
Sep 17 18:26:52.496280 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496268    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="storage-initializer"
Sep 17 18:26:52.496334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496281    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="storage-initializer"
Sep 17 18:26:52.496334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496295    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container"
Sep 17 18:26:52.496334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container"
Sep 17 18:26:52.496334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496313    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kube-rbac-proxy"
Sep 17 18:26:52.496334 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kube-rbac-proxy"
Sep 17 18:26:52.496550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496380    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kserve-container"
Sep 17 18:26:52.496550 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.496388    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="925ad428-c9ad-4e66-a9f8-a1bc08255d5b" containerName="kube-rbac-proxy"
Sep 17 18:26:52.498587 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.498569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.501197 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.501167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:26:52.501329 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.501203    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:26:52.501779 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.501762    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:26:52.501915 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.501777    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:26:52.501915 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.501765    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 18:26:52.501915 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.501824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:26:52.512658 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.512631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"]
Sep 17 18:26:52.552070 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.552039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:52.587240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.587142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93396a7f-1540-4e11-a056-a126411ab972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.587240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.587192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfv7k\" (UniqueName: \"kubernetes.io/projected/93396a7f-1540-4e11-a056-a126411ab972-kube-api-access-pfv7k\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.587240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.587211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93396a7f-1540-4e11-a056-a126411ab972-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.587475 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.587323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93396a7f-1540-4e11-a056-a126411ab972-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.688623 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.688580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93396a7f-1540-4e11-a056-a126411ab972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.688623 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.688626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfv7k\" (UniqueName: \"kubernetes.io/projected/93396a7f-1540-4e11-a056-a126411ab972-kube-api-access-pfv7k\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.688879 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.688645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93396a7f-1540-4e11-a056-a126411ab972-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.688879 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.688687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93396a7f-1540-4e11-a056-a126411ab972-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.689186 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.689164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93396a7f-1540-4e11-a056-a126411ab972-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.689475 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.689454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93396a7f-1540-4e11-a056-a126411ab972-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.691321 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.691300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93396a7f-1540-4e11-a056-a126411ab972-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.697144 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.697116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfv7k\" (UniqueName: \"kubernetes.io/projected/93396a7f-1540-4e11-a056-a126411ab972-kube-api-access-pfv7k\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:52.810598 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:52.810562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:26:53.086302 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:53.086240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"]
Sep 17 18:26:53.088687 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:26:53.088657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93396a7f_1540_4e11_a056_a126411ab972.slice/crio-5dc65c1fe6b4fb0f74959c1bb48b459d8a342e6dccf10eff99fb94d7813a4d85 WatchSource:0}: Error finding container 5dc65c1fe6b4fb0f74959c1bb48b459d8a342e6dccf10eff99fb94d7813a4d85: Status 404 returned error can't find the container with id 5dc65c1fe6b4fb0f74959c1bb48b459d8a342e6dccf10eff99fb94d7813a4d85
Sep 17 18:26:54.028848 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:54.028807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" event={"ID":"93396a7f-1540-4e11-a056-a126411ab972","Type":"ContainerStarted","Data":"c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb"}
Sep 17 18:26:54.029328 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:54.028881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" event={"ID":"93396a7f-1540-4e11-a056-a126411ab972","Type":"ContainerStarted","Data":"5dc65c1fe6b4fb0f74959c1bb48b459d8a342e6dccf10eff99fb94d7813a4d85"}
Sep 17 18:26:59.053603 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:59.053576    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99_93396a7f-1540-4e11-a056-a126411ab972/storage-initializer/0.log"
Sep 17 18:26:59.054030 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:59.053617    2563 generic.go:356] "Generic (PLEG): container finished" podID="93396a7f-1540-4e11-a056-a126411ab972" containerID="c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb" exitCode=1
Sep 17 18:26:59.054030 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:26:59.053689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" event={"ID":"93396a7f-1540-4e11-a056-a126411ab972","Type":"ContainerDied","Data":"c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb"}
Sep 17 18:27:00.059553 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:00.059526    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99_93396a7f-1540-4e11-a056-a126411ab972/storage-initializer/0.log"
Sep 17 18:27:00.060006 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:00.059607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" event={"ID":"93396a7f-1540-4e11-a056-a126411ab972","Type":"ContainerStarted","Data":"1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45"}
Sep 17 18:27:02.475442 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:02.475402    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"]
Sep 17 18:27:02.475900 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:02.475675    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer" containerID="cri-o://1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45" gracePeriod=30
Sep 17 18:27:02.496437 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:02.496388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:03.631205 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.631169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:03.710873 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.710837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:03.823817 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.823771    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99_93396a7f-1540-4e11-a056-a126411ab972/storage-initializer/1.log"
Sep 17 18:27:03.824193 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.824176    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99_93396a7f-1540-4e11-a056-a126411ab972/storage-initializer/0.log"
Sep 17 18:27:03.825482 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.825463    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:27:03.995147 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.995110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfv7k\" (UniqueName: \"kubernetes.io/projected/93396a7f-1540-4e11-a056-a126411ab972-kube-api-access-pfv7k\") pod \"93396a7f-1540-4e11-a056-a126411ab972\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") "
Sep 17 18:27:03.995338 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.995201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93396a7f-1540-4e11-a056-a126411ab972-proxy-tls\") pod \"93396a7f-1540-4e11-a056-a126411ab972\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") "
Sep 17 18:27:03.995338 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.995222    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93396a7f-1540-4e11-a056-a126411ab972-kube-rbac-proxy-sar-config\") pod \"93396a7f-1540-4e11-a056-a126411ab972\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") "
Sep 17 18:27:03.995338 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.995265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93396a7f-1540-4e11-a056-a126411ab972-kserve-provision-location\") pod \"93396a7f-1540-4e11-a056-a126411ab972\" (UID: \"93396a7f-1540-4e11-a056-a126411ab972\") "
Sep 17 18:27:03.995589 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.995564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93396a7f-1540-4e11-a056-a126411ab972-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93396a7f-1540-4e11-a056-a126411ab972" (UID: "93396a7f-1540-4e11-a056-a126411ab972"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:03.995650 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.995620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93396a7f-1540-4e11-a056-a126411ab972-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93396a7f-1540-4e11-a056-a126411ab972" (UID: "93396a7f-1540-4e11-a056-a126411ab972"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:03.997203 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.997182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93396a7f-1540-4e11-a056-a126411ab972-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93396a7f-1540-4e11-a056-a126411ab972" (UID: "93396a7f-1540-4e11-a056-a126411ab972"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:03.997284 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:03.997268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93396a7f-1540-4e11-a056-a126411ab972-kube-api-access-pfv7k" (OuterVolumeSpecName: "kube-api-access-pfv7k") pod "93396a7f-1540-4e11-a056-a126411ab972" (UID: "93396a7f-1540-4e11-a056-a126411ab972"). InnerVolumeSpecName "kube-api-access-pfv7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:04.078439 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.078405    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99_93396a7f-1540-4e11-a056-a126411ab972/storage-initializer/1.log"
Sep 17 18:27:04.078875 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.078859    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99_93396a7f-1540-4e11-a056-a126411ab972/storage-initializer/0.log"
Sep 17 18:27:04.078946 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.078895    2563 generic.go:356] "Generic (PLEG): container finished" podID="93396a7f-1540-4e11-a056-a126411ab972" containerID="1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45" exitCode=1
Sep 17 18:27:04.079010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.078941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" event={"ID":"93396a7f-1540-4e11-a056-a126411ab972","Type":"ContainerDied","Data":"1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45"}
Sep 17 18:27:04.079010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.078957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"
Sep 17 18:27:04.079010 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.078997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99" event={"ID":"93396a7f-1540-4e11-a056-a126411ab972","Type":"ContainerDied","Data":"5dc65c1fe6b4fb0f74959c1bb48b459d8a342e6dccf10eff99fb94d7813a4d85"}
Sep 17 18:27:04.079113 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.079023    2563 scope.go:117] "RemoveContainer" containerID="1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45"
Sep 17 18:27:04.090903 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.090874    2563 scope.go:117] "RemoveContainer" containerID="c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb"
Sep 17 18:27:04.096035 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.096007    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfv7k\" (UniqueName: \"kubernetes.io/projected/93396a7f-1540-4e11-a056-a126411ab972-kube-api-access-pfv7k\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:27:04.096141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.096038    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93396a7f-1540-4e11-a056-a126411ab972-proxy-tls\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:27:04.096141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.096053    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93396a7f-1540-4e11-a056-a126411ab972-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:27:04.096141 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.096066    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93396a7f-1540-4e11-a056-a126411ab972-kserve-provision-location\") on node \"ip-10-0-135-202.ec2.internal\" DevicePath \"\""
Sep 17 18:27:04.099166 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.099137    2563 scope.go:117] "RemoveContainer" containerID="1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45"
Sep 17 18:27:04.099476 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:27:04.099458    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45\": container with ID starting with 1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45 not found: ID does not exist" containerID="1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45"
Sep 17 18:27:04.099540 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.099485    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45"} err="failed to get container status \"1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45\": rpc error: code = NotFound desc = could not find container \"1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45\": container with ID starting with 1bb08ad94ff97d543dd4b1559b51ce744577ad41d9fe0809f35ed5660aedcb45 not found: ID does not exist"
Sep 17 18:27:04.099540 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.099505    2563 scope.go:117] "RemoveContainer" containerID="c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb"
Sep 17 18:27:04.099749 ip-10-0-135-202 kubenswrapper[2563]: E0917 18:27:04.099734    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb\": container with ID starting with c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb not found: ID does not exist" containerID="c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb"
Sep 17 18:27:04.099811 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.099754    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb"} err="failed to get container status \"c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb\": rpc error: code = NotFound desc = could not find container \"c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb\": container with ID starting with c454d99872993d0b539476ea56bfe76db6eda4633194a6b692045309b196ebbb not found: ID does not exist"
Sep 17 18:27:04.130885 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.130850    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"]
Sep 17 18:27:04.139128 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.139094    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-f9c6f8ddc-wsd99"]
Sep 17 18:27:04.959916 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:04.959887    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93396a7f-1540-4e11-a056-a126411ab972" path="/var/lib/kubelet/pods/93396a7f-1540-4e11-a056-a126411ab972/volumes"
Sep 17 18:27:11.049382 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:11.049344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:15.727530 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:15.727497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:20.855392 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:20.855356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:20.955547 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:20.955505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:26.426936 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:27:26.426899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:22.611341 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:28:22.611301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:27.772377 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:28:27.772331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:27.939645 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:28:27.939604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:55.275584 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:28:55.275541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:00.409695 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:29:00.409666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:00.835855 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:29:00.835819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:21.469854 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:29:21.469753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:21.570712 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:29:21.570658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.278480 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:29:57.278436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.347155 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:29:57.347120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.609031 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:09.608996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.077528 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:26.077491    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:33.425651 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:33.425620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:30:33.434024 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:33.434004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:30:48.847562 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:48.847475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.133865 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:52.133809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.232036 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:57.232000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.307104 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:30:57.307070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:43.497519 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:32:43.497437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:44.912963 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:32:44.912932    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:00.541462 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:00.541417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:00.606348 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:00.606315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:06.986768 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:06.986726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:07.087949 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:07.087914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:38.747601 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:38.747513    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:43.515544 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:43.515507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:43.671108 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:43.671064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:43.736924 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:43.736891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:46.786147 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:46.786119    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sd6ff_05a68412-83e9-486b-a686-65730e2448ba/global-pull-secret-syncer/0.log"
Sep 17 18:33:46.890437 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:46.890408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log"
Sep 17 18:33:47.047817 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:47.047733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-202.ec2.internal_a8870edaf0c4a8354936a672f9fccc76/haproxy/0.log"
Sep 17 18:33:49.561380 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:49.561338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:49.629108 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:49.629074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-btktp_1aacb1a6-3867-4f18-8241-4d9e294d8db0/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:50.525433 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.525403    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/alertmanager/0.log"
Sep 17 18:33:50.580983 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.580958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/config-reloader/0.log"
Sep 17 18:33:50.626810 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.626764    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/kube-rbac-proxy-web/0.log"
Sep 17 18:33:50.666351 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.666325    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/kube-rbac-proxy/0.log"
Sep 17 18:33:50.713926 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.713903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/kube-rbac-proxy-metric/0.log"
Sep 17 18:33:50.754211 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.754188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/prom-label-proxy/0.log"
Sep 17 18:33:50.800637 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.800566    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd8e401e-106b-4959-a7d9-76ae8b9c4b4a/init-config-reloader/0.log"
Sep 17 18:33:50.907497 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.907468    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-rpqc9_17e59f32-2f84-4564-911d-a92a741ce373/kube-state-metrics/0.log"
Sep 17 18:33:50.944602 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.944575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-rpqc9_17e59f32-2f84-4564-911d-a92a741ce373/kube-rbac-proxy-main/0.log"
Sep 17 18:33:50.975391 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:50.975357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-rpqc9_17e59f32-2f84-4564-911d-a92a741ce373/kube-rbac-proxy-self/0.log"
Sep 17 18:33:51.181529 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.181505    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cz4rq_5d315ace-900d-42d6-960d-cdac601e70b5/node-exporter/0.log"
Sep 17 18:33:51.226032 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.226006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cz4rq_5d315ace-900d-42d6-960d-cdac601e70b5/kube-rbac-proxy/0.log"
Sep 17 18:33:51.305223 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.305201    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cz4rq_5d315ace-900d-42d6-960d-cdac601e70b5/init-textfile/0.log"
Sep 17 18:33:51.872507 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.872475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-556cd5b59-pm54b_f68a0ad2-254b-43b9-809b-da9cfb8ee1e0/telemeter-client/0.log"
Sep 17 18:33:51.892694 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.892668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-556cd5b59-pm54b_f68a0ad2-254b-43b9-809b-da9cfb8ee1e0/reload/0.log"
Sep 17 18:33:51.916102 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.916075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-556cd5b59-pm54b_f68a0ad2-254b-43b9-809b-da9cfb8ee1e0/kube-rbac-proxy/0.log"
Sep 17 18:33:51.951341 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.951324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c88cf684-q7w5t_6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23/thanos-query/0.log"
Sep 17 18:33:51.970491 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.970468    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c88cf684-q7w5t_6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23/kube-rbac-proxy-web/0.log"
Sep 17 18:33:51.991815 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:51.991786    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c88cf684-q7w5t_6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23/kube-rbac-proxy/0.log"
Sep 17 18:33:52.020240 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:52.020222    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c88cf684-q7w5t_6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23/prom-label-proxy/0.log"
Sep 17 18:33:52.035752 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:52.035728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c88cf684-q7w5t_6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23/kube-rbac-proxy-rules/0.log"
Sep 17 18:33:52.056006 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:52.055987    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c88cf684-q7w5t_6f7ad3e3-d2a8-47f5-97e1-2d3858bb6d23/kube-rbac-proxy-metrics/0.log"
Sep 17 18:33:53.959654 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:53.959623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/1.log"
Sep 17 18:33:53.967480 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:53.967447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-p9xpb_4e52a62f-f2fa-4b66-be91-758b8dffcfbd/console-operator/2.log"
Sep 17 18:33:54.389236 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.389208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-868fdb6c4c-8p26w_1f6b9919-8efd-4253-aab7-0cce002ad3c9/console/0.log"
Sep 17 18:33:54.428735 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.428710    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-p62zj_40cb78ac-a56b-49f3-805c-1ab809d43d4c/download-server/0.log"
Sep 17 18:33:54.960787 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.960755    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"]
Sep 17 18:33:54.961318 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.961299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer"
Sep 17 18:33:54.961359 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.961322    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer"
Sep 17 18:33:54.961395 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.961353    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer"
Sep 17 18:33:54.961395 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.961382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer"
Sep 17 18:33:54.961485 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.961474    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer"
Sep 17 18:33:54.961520 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.961489    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93396a7f-1540-4e11-a056-a126411ab972" containerName="storage-initializer"
Sep 17 18:33:54.965152 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.965130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:54.967351 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.967316    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4nv9\"/\"openshift-service-ca.crt\""
Sep 17 18:33:54.967456 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.967382    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m4nv9\"/\"default-dockercfg-vxkht\""
Sep 17 18:33:54.968053 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.968038    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4nv9\"/\"kube-root-ca.crt\""
Sep 17 18:33:54.976305 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:54.976285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"]
Sep 17 18:33:55.114444 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.114415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-podres\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.114597 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.114459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmmr\" (UniqueName: \"kubernetes.io/projected/faa34c3a-998f-4ba9-a193-0fec35558c76-kube-api-access-9wmmr\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.114597 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.114476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-proc\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.114597 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.114490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-lib-modules\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.114597 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.114591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-sys\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.215935 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.215868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-sys\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.215935 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.215903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-podres\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.215935 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.215932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmmr\" (UniqueName: \"kubernetes.io/projected/faa34c3a-998f-4ba9-a193-0fec35558c76-kube-api-access-9wmmr\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.216148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.215947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-proc\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.216148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.215961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-lib-modules\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.216148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.215990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-sys\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.216148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.216034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-podres\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.216148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.216035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-proc\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.216148 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.216065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/faa34c3a-998f-4ba9-a193-0fec35558c76-lib-modules\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.224107 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.224082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmmr\" (UniqueName: \"kubernetes.io/projected/faa34c3a-998f-4ba9-a193-0fec35558c76-kube-api-access-9wmmr\") pod \"perf-node-gather-daemonset-4hxvc\" (UID: \"faa34c3a-998f-4ba9-a193-0fec35558c76\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.275867 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.275848    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:55.551853 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.551828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"]
Sep 17 18:33:55.554268 ip-10-0-135-202 kubenswrapper[2563]: W0917 18:33:55.554227    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfaa34c3a_998f_4ba9_a193_0fec35558c76.slice/crio-75dbe8977c4b7a59be63b5dae2ba34cd8d023ed005c6e94259bba48509c047ee WatchSource:0}: Error finding container 75dbe8977c4b7a59be63b5dae2ba34cd8d023ed005c6e94259bba48509c047ee: Status 404 returned error can't find the container with id 75dbe8977c4b7a59be63b5dae2ba34cd8d023ed005c6e94259bba48509c047ee
Sep 17 18:33:55.996926 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.996894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc" event={"ID":"faa34c3a-998f-4ba9-a193-0fec35558c76","Type":"ContainerStarted","Data":"123c44ee0643acf620e09d22c5d9237f058ec85b4cfdd56e858e7aa82dcf860a"}
Sep 17 18:33:55.997281 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.996946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc" event={"ID":"faa34c3a-998f-4ba9-a193-0fec35558c76","Type":"ContainerStarted","Data":"75dbe8977c4b7a59be63b5dae2ba34cd8d023ed005c6e94259bba48509c047ee"}
Sep 17 18:33:55.997281 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:55.996978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:33:56.013922 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:56.013872    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc" podStartSLOduration=2.013858606 podStartE2EDuration="2.013858606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:56.011641785 +0000 UTC m=+3521.539423611" watchObservedRunningTime="2026-09-17 18:33:56.013858606 +0000 UTC m=+3521.541640567"
Sep 17 18:33:56.030955 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:56.030932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmsks_9db7af89-b3ff-44d8-8354-525579dcdf5b/dns/0.log"
Sep 17 18:33:56.049578 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:56.049551    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmsks_9db7af89-b3ff-44d8-8354-525579dcdf5b/kube-rbac-proxy/0.log"
Sep 17 18:33:56.091241 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:56.091219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lwf8b_6db84866-0cdb-4b78-b53d-5c019281e212/dns-node-resolver/0.log"
Sep 17 18:33:56.604880 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:56.604845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t5jhz_4338b764-2fc9-42cf-a3a2-61f47006dbbd/node-ca/0.log"
Sep 17 18:33:57.370489 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:57.370461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56879d8b98-cgl46_6abc45d9-b9e0-4f47-9819-97ad80b332ee/router/0.log"
Sep 17 18:33:57.698112 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:57.698082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jczs6_4622bbec-ed20-4923-a475-1e138ed7b050/serve-healthcheck-canary/0.log"
Sep 17 18:33:58.221731 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:58.221697    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxzlk_38a82660-d31e-4922-a63e-98d2ccf89243/kube-rbac-proxy/0.log"
Sep 17 18:33:58.241083 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:58.241058    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxzlk_38a82660-d31e-4922-a63e-98d2ccf89243/exporter/0.log"
Sep 17 18:33:58.262111 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:33:58.262086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pxzlk_38a82660-d31e-4922-a63e-98d2ccf89243/extractor/0.log"
Sep 17 18:34:00.767661 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:00.767624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-kz528_7802228e-f1b0-4892-84fb-18dae4f656b1/manager/0.log"
Sep 17 18:34:01.261115 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:01.261082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-hmvmf_99467336-5c7e-4ff8-a159-4aa4d0b02150/manager/0.log"
Sep 17 18:34:01.451870 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:01.451841    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-th4xh_4e70ba67-b63a-42f4-b552-ee8926dadfb8/seaweedfs/0.log"
Sep 17 18:34:01.476117 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:01.476086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-2ldws_a30e69a3-1217-4da7-8eab-59cdf49c59d0/seaweedfs-tls-custom/0.log"
Sep 17 18:34:02.011222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:02.011191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-4hxvc"
Sep 17 18:34:06.981784 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:06.981759    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:34:07.002538 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.002502    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/egress-router-binary-copy/0.log"
Sep 17 18:34:07.022023 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.022003    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/cni-plugins/0.log"
Sep 17 18:34:07.042136 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.042113    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/bond-cni-plugin/0.log"
Sep 17 18:34:07.062547 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.062522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/routeoverride-cni/0.log"
Sep 17 18:34:07.083071 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.083051    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/whereabouts-cni-bincopy/0.log"
Sep 17 18:34:07.107717 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.107695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-knj9c_2d5c05eb-c75a-45ad-8f2a-06ea33b0005d/whereabouts-cni/0.log"
Sep 17 18:34:07.589054 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.589006    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-chk69_4d3b541d-198a-49f5-be28-85d0ea5f1c88/kube-multus/0.log"
Sep 17 18:34:07.787222 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.787188    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zxjft_d2d139ed-fedb-42f4-b2d7-10e810b16090/network-metrics-daemon/0.log"
Sep 17 18:34:07.805423 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:07.805384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zxjft_d2d139ed-fedb-42f4-b2d7-10e810b16090/kube-rbac-proxy/0.log"
Sep 17 18:34:08.704593 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.704549    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/ovn-controller/0.log"
Sep 17 18:34:08.725864 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.725835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/ovn-acl-logging/0.log"
Sep 17 18:34:08.767300 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.767271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/kube-rbac-proxy-node/0.log"
Sep 17 18:34:08.804392 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.804372    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:34:08.822768 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.822737    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/northd/0.log"
Sep 17 18:34:08.864064 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.864046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/nbdb/0.log"
Sep 17 18:34:08.887724 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.887692    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/sbdb/0.log"
Sep 17 18:34:08.963946 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:08.963883    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmbb8_ecef2930-3fcc-4dbb-b497-f58796174663/ovnkube-controller/0.log"
Sep 17 18:34:10.507614 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:10.507584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7cbc45fddc-np47h_9457fa88-f119-457f-82bd-e3aaa296c5f3/check-endpoints/0.log"
Sep 17 18:34:10.573252 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:10.573212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7swdh_b206d4e1-1b71-457b-b416-a6d0dfdf36a2/network-check-target-container/0.log"
Sep 17 18:34:11.668085 ip-10-0-135-202 kubenswrapper[2563]: I0917 18:34:11.668049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bxb95_3921a5cf-6472-434c-a537-f1658c67a4d1/iptables-alerter/0.log"